> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Kandinsky6 > Deploy Kandinsky 6.0 for joint text- and image-conditioned video and audio generation. export const config = { modelName: "Kandinsky 6.0", supportedHardware: ["b200", "gb300", "rtxpro6000"], hardware: [{ id: "rtxpro6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" }], groupHardware: false, matchDims: [], overlayDims: [{ id: "weights", title: "Checkpoint", scope: "base", default: "distill", description: "Pro-distill uses 10 steps without CFG; Pro uses 50 steps with CFG.", options: [{ id: "distill", label: "Pro-distill", recommended: true }, { id: "pro", label: "Pro" }] }, { id: "mode", title: "Conditioning", scope: "base", default: "text", description: "Both modes generate video and audio through the same server.", options: [{ id: "text", label: "Text" }, { id: "image", label: "Text and image" }] }, { id: "attention", title: "Attention", scope: "serve", default: "platform", description: "The recipe uses FA on B200/GB300 and SDPA on RTX PRO 6000. Sage3 overrides only the DiT; kernel changes can alter both outputs.", learnMore: "#rtx-pro-6000-and-sageattention3", options: [{ id: "platform", label: "Recipe default", recommended: true }, { id: "fa", label: "FlashAttention", disabled: s => s.hw === "rtxpro6000", disableReason: "This SM120 platform maps FA to SDPA; select SDPA explicitly instead." }, { id: "torch_sdpa", label: "PyTorch SDPA", soft: s => s.hw !== "rtxpro6000", softReason: "Full-checkpoint SDPA validation used RTX PRO 6000.", disabled: s => s.topology_mode === "manual" && Number(s.ring_degree) > 1, disableReason: "SDPA does not support Ring rotation." }, { id: "sage_attn_3", label: "Sage3 (approximate)", soft: true, disabled: s => s.hw !== "rtxpro6000", disableReason: "The tested Sage3 build requires SM120/121; it rejects B200/GB300.", softReason: "Offline benchmark only; this HTTP recipe is unverified. Requires a separate SageAttention3 installation.", description: "RTX: 21.3% lower latency, but video PSNR 11.97 dB and audio SNR -2.21 dB versus SDPA. Not lossless or quality-certified.", hints: ["Sage3 is approximate and changes video/audio substantially. Install the documented SM120 SageAttention3 build first."] }] }, { id: "placement", title: "Weight placement", scope: "serve", default: "resident", description: "Keep weights resident when they fit; trade transfer time for lower GPU memory otherwise.", learnMore: "#choosing-a-topology", options: [{ id: "resident", label: "Resident", recommended: true, description: "Baseline arithmetic, without weight transfers between stages." }, { id: "fsdp", label: "FSDP", description: "Shards DiT weights. GB300 Ulysses4 matched resident RGB/audio, using less GPU memory at higher latency.", flags: ["--use-fsdp-inference true"], disabled: s => Number(s.gpus_per_node) < 2, disableReason: "Weight sharding requires multiple GPUs." }, { id: "component", label: "Component offload", soft: true, description: "Transfers selected components whole around their use. Partial recipes need their own memory and latency validation.", flags: s => [`--cpu-offload-components ${s.offload_components}`] }, { id: "offload", label: "Layerwise offload", description: "Streams blocks from host RAM. This adds transfers and pinned host memory; not a free speedup.", flags: s => [`--layerwise-offload-components ${s.offload_components}`] }] }, { id: "offload_components", title: "Offload components", scope: "serve", default: "all", showWhen: s => ["component", "offload"].includes(s.placement), description: "Text encoders includes both encoders; VAEs includes video and audio. Partial selections are unverified recipes.", learnMore: "#choosing-a-topology", options: [{ id: "all", label: "All components" }, { id: "transformer", label: "DiT", soft: true }, { id: "text_encoder", label: "Text encoders", soft: true }, { id: "vae audio_vae", label: "VAEs", soft: true }, { id: "audio_vae", label: "Audio VAE", soft: true }, { id: "text_encoder vae audio_vae", label: "Encoders + VAEs", soft: true }] }, { id: "prefetch", title: "DiT prefetch", scope: "serve", kind: "number", min: 1, max: 4, default: 1, unit: "layers", options: [], showWhen: s => s.placement === "offload" && ["all", "transformer"].includes(s.offload_components), verifiedWhen: s => Number(s.prefetch) === 1, description: "More prefetched layers use more VRAM and may overlap transfers better. Values above 1 are unverified here.", learnMore: "#choosing-a-topology" }, { id: "resident_layers", title: "DiT resident layers", scope: "serve", kind: "number", min: 0, max: 8, default: 0, unit: "layers", options: [], showWhen: s => s.placement === "offload" && ["all", "transformer"].includes(s.offload_components), verifiedWhen: s => Number(s.resident_layers) === 0, description: "Keep leading DiT layers on GPU during the request. Trades VRAM for fewer transfers; nonzero recipes are unverified.", learnMore: "#choosing-a-topology" }, { id: "workload", title: "Workload", scope: "request", default: "full", description: "Use the lightweight request to check deployment, not output quality.", options: [{ id: "full", label: "768 x 512, 121 frames", recommended: true }, { id: "smoke", label: "384 x 256, 17 frames, 2 steps" }] }, { id: "cache", title: "Cache-DiT", scope: "request", default: "off", description: "Request-level block reuse can change both video and audio. The default cache policy is not a validated quality preset.", learnMore: "#cache-dit", options: [{ id: "off", label: "Off", recommended: true }, { id: "on", label: "On (approximate)", soft: true, description: "Uses the runtime cache defaults. Short requests may finish before cache warmup ends, with no acceleration." }] }], commandBuilder: { defaultSelection: { hw: "b200", nodes: 1, gpus_per_node: 1, topology_mode: "auto", tp_size: 1, ulysses_degree: 1, ring_degree: 1 }, resource: { limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 4 } }, verifiedRecipes: [["b200", 1, 1], ["b200", 2, 1], ["gb300", 4, 1], ["gb300", 4, 2], ["rtxpro6000", 1, 1]].map(([hw, gpus, tp]) => ({ id: `${hw}-distill-${gpus}-tp${tp}`, hw, weights: "distill", placement: "resident", nodes: 1, gpus_per_node: gpus, tp_size: tp, ulysses_degree: gpus / tp, ring_degree: 1, default: tp === 1 && (hw === "gb300" ? gpus === 4 : gpus === 1) })).concat([{ id: "gb300-distill-fsdp", hw: "gb300", weights: "distill", placement: "fsdp", nodes: 1, gpus_per_node: 4, tp_size: 1, ulysses_degree: 4, ring_degree: 1 }]), autoTopology: s => ({ tp_size: 1, ulysses_degree: Number(s.gpus_per_node), ring_degree: 1 }), validateTopology: (s, t) => { const errors = []; const degrees = [t.tp_size, t.ulysses_degree, t.ring_degree]; if (!degrees.every(n => Number.isInteger(n) && n >= 1)) errors.push("Parallel degrees must be positive integers."); if (Number(s.nodes) !== 1) errors.push("This picker covers one node."); if (degrees.reduce((a, b) => a * b, 1) !== Number(s.gpus_per_node)) errors.push("GPU count must equal TP times Ulysses times Ring."); if (32 % (t.tp_size * t.ulysses_degree)) errors.push("TP times Ulysses must divide the 32 DiT heads."); if (s.placement === "fsdp" && Number(s.gpus_per_node) < 2) errors.push("FSDP requires multiple GPUs."); if ((["torch_sdpa", "sage_attn_3"].includes(s.attention) || s.hw === "rtxpro6000") && t.ring_degree > 1) errors.push("SDPA and Sage3 do not support Ring rotation; use FlashAttention on B200/GB300."); if (s.attention === "fa" && s.hw === "rtxpro6000") errors.push("FA resolves to SDPA on this SM120 platform; select SDPA instead."); if (s.attention === "sage_attn_3" && s.hw !== "rtxpro6000") errors.push("The tested Sage3 build requires SM120/121."); return errors; } }, resolveDeployment: s => { const resource = config.commandBuilder.resource; const topology = s.topology_mode === "manual" ? { tp_size: Number(s.tp_size), ulysses_degree: Number(s.ulysses_degree), ring_degree: Number(s.ring_degree) } : resource.autoTopology(s); const errors = resource.validateTopology(s, topology); const platformAttention = s.hw === "rtxpro6000" ? "torch_sdpa" : "fa"; const attention = s.attention === "platform" ? platformAttention : s.attention; const streamingDit = s.placement === "offload" && ["all", "transformer"].includes(s.offload_components); const recipe = resource.verifiedRecipes.find(r => r.hw === s.hw && r.weights === s.weights && r.placement === s.placement && r.nodes === Number(s.nodes) && r.gpus_per_node === Number(s.gpus_per_node) && r.tp_size === topology.tp_size && r.ulysses_degree === topology.ulysses_degree && r.ring_degree === topology.ring_degree); const served = !!recipe && errors.length === 0 && attention === platformAttention; const requested = served && s.workload === "smoke" && s.cache === "off"; const warnings = []; if (!requested) warnings.push(served ? "Only the uncached lightweight workload is verified for this HTTP recipe; full-size generation is checked offline." : "This exact checkpoint, placement and topology combination has not completed HTTP verification."); if (topology.tp_size > 1) warnings.push("TP changes video and audio outputs. Prefer Ulysses when memory permits; validate quality for your workload."); if (topology.ring_degree > 1) warnings.push("Ring changes floating-point reductions; bitwise video/audio parity is not established."); if (["component", "offload"].includes(s.placement)) warnings.push("Offload needs additional host RAM and transfer bandwidth; DiT prefetch and resident layers also consume VRAM."); if (attention !== platformAttention) warnings.push("This attention override has not completed the exact HTTP recipe; changing kernels can change video and audio."); if (attention === "sage_attn_3") warnings.push("Sage3 is approximate: measured video PSNR 11.97 dB and audio SNR -2.21 dB versus SDPA; no quality guarantee."); if (s.cache === "on") warnings.push("Cache-DiT is approximate and affects both streams; validate video and audio. Short requests may not skip any blocks."); const model = s.weights === "pro" ? "kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers" : "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers"; return { match: { hw: s.hw }, nnodes: Number(s.nodes), verified: requested, flags: [`--model-path ${model}`, `--num-gpus ${Number(s.gpus_per_node)}`, `--tp-size ${topology.tp_size}`, `--ulysses-degree ${topology.ulysses_degree}`, `--ring-degree ${topology.ring_degree}`, "--performance-mode speed", `--attention-backend ${attention === "sage_attn_3" ? platformAttention : attention}`, ...attention === "sage_attn_3" ? ["--component-attention-backends transformer=sage_attn_3"] : [], "--warmup-mode off", "--host {{HOST_IP}}", "--port {{PORT}}", ...streamingDit && Number(s.prefetch) !== 1 ? [`--dit-offload-prefetch-size ${s.prefetch}`] : [], ...streamingDit && Number(s.resident_layers) !== 0 ? [`--dit-layerwise-resident-layers ${s.resident_layers}`] : []], builder: { topology, resolvedSettings: { attention: ({ fa: "FlashAttention", torch_sdpa: "PyTorch SDPA", sage_attn_3: "Sage3 (approximate)" })[attention], prefetch: `${s.prefetch} ${Number(s.prefetch) === 1 ? "layer" : "layers"}`, resident_layers: `${s.resident_layers} ${Number(s.resident_layers) === 1 ? "layer" : "layers"}` }, topologySummary: `TP ${topology.tp_size}, Ulysses ${topology.ulysses_degree}, Ring ${topology.ring_degree}`, errors, warnings, verification: { serve: errors.length ? "error" : served ? "verified" : "unverified", request: errors.length ? "error" : requested ? "verified" : "unverified" } } }; } }, modelNames: { default: "kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers" }, placeholders: { HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, PORT: { target: "command", label: "Bind port", default: "30000" }, CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, INPUT_IMAGE: { target: "curl", label: "Reference PNG", default: "/path/to/reference.png" } }, curl: s => { const smoke = s.workload === "smoke"; const body = { prompt: "A woman chops vegetables in a sunlit kitchen, with soft jazz playing.", size: smoke ? "384x256" : "768x512", num_frames: smoke ? 17 : 121, fps: 24, seed: 42, enable_cache_dit: s.cache === "on" }; if (smoke) body.num_inference_steps = 2; if (s.mode === "text") return `curl -sS --fail-with-body http://{{CURL_HOST}}:{{CURL_PORT}}/v1/videos \\ -H 'Content-Type: application/json' \\ -d '${JSON.stringify(body, null, 2)}'`; const fields = Object.entries(body).map(([key, value]) => ` --form-string '${key}=${value}'`); fields.push(' -F "input_reference=@{{INPUT_IMAGE}};type=image/png"'); return `curl -sS --fail-with-body http://{{CURL_HOST}}:{{CURL_PORT}}/v1/videos \\ ${fields.join(" \\\n")}`; }, runModes: () => ["python"], showPlaygroundLink: false, cells: [] }; export const Deployment = ({config, benchmarks}) => { if (!config) { return
Deployment: missing config prop
; } const AMD_RDMA_DOCKER_FLAGS = ["--device /dev/infiniband", "--cap-add IPC_LOCK", "--ulimit memlock=-1", "--ulimit stack=67108864", "--ulimit nofile=1048576:1048576"]; const HARDWARE_CATALOG = { blackwell: [{ id: "b300", label: "B300", vram: "288GB" }, { id: "gb300", label: "GB300", vram: "288GB" }, { id: "b200", label: "B200", vram: "192GB" }, { id: "gb200", label: "GB200", vram: "192GB" }, { id: "dgx-spark", label: "DGX Spark", vram: "128GB", multiNodeDockerFlags: ["--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband"] }], hopper: [{ id: "h200", label: "H200", vram: "141GB" }, { id: "h100", label: "H100", vram: "80GB" }, { id: "h20-3e", label: "H20-3e", vram: "141GB" }, { id: "h800", label: "H800", vram: "80GB" }], amd: [{ id: "mi300x", label: "MI300X", vram: "192GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi325x", label: "MI325X", vram: "256GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi350x", label: "MI350X", vram: "288GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi355x", label: "MI355X", vram: "288GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }], npu: [{ id: "a3", label: "A3 Series", vram: "64GB/die", npuDevices: 16 }, { id: "a5", label: "950PR/DT Series", vram: "128GB", npuDevices: 8 }] }; const makeStyles = isDark => ({ container: { maxWidth: "900px", margin: "0 auto", display: "flex", flexDirection: "column", gap: "3px" }, card: { padding: "5px 10px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", display: "flex", alignItems: "center", gap: "10px", background: isDark ? "#1f2937" : "#fff" }, cardColumn: { padding: "5px 10px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", display: "flex", flexDirection: "column", gap: "4px", background: isDark ? "#1f2937" : "#fff" }, title: { fontSize: "12px", fontWeight: "600", minWidth: "108px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" }, vendorRow: { display: "flex", alignItems: "center", gap: "6px" }, vendorLabel: { fontSize: "10px", fontWeight: "600", color: isDark ? "#9ca3af" : "#6b7280", width: "68px", flexShrink: 0, textTransform: "uppercase", letterSpacing: "0.04em" }, itemsGrid: () => ({ display: "grid", gridTemplateColumns: "repeat(auto-fit, minmax(72px, 1fr))", gap: "4px", flex: 1 }), labelBase: { padding: "2px 8px", border: `1px solid ${isDark ? "#9ca3af" : "#d1d5db"}`, borderRadius: "3px", cursor: "pointer", display: "inline-flex", flexDirection: "column", alignItems: "center", justifyContent: "center", fontWeight: "500", fontSize: "12px", transition: "all 0.2s", userSelect: "none", minHeight: "26px", textAlign: "center", background: isDark ? "#374151" : "#fff", color: isDark ? "#e5e7eb" : "inherit" }, checked: { background: "#D45D44", color: "white", borderColor: "#D45D44" }, disabled: { cursor: "not-allowed", opacity: 0.4 }, subtitle: { display: "block", fontSize: "9px", marginTop: "1px", lineHeight: "1.1", opacity: 0.7 }, commandWrap: { position: "relative", flex: 1, background: isDark ? "#111827" : "#f5f5f5", borderRadius: "6px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, overflow: "hidden" }, commandHeader: { display: "flex", flexWrap: "wrap", justifyContent: "space-between", alignItems: "center", gap: "6px 10px", padding: "6px 10px", borderBottom: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, background: isDark ? "#1f2937" : "#fafafa" }, commandPre: { padding: "12px 16px", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontSize: "12px", lineHeight: "1.5", color: isDark ? "#e5e7eb" : "#374151", whiteSpace: "pre-wrap", overflowX: "auto", margin: 0 }, mtpWarn: { margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px", fontSize: "12px", lineHeight: "1.45", background: isDark ? "#78350f" : "#fef3c7", color: isDark ? "#fde68a" : "#92400e", border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}` }, badge: status => ({ display: "inline-flex", alignItems: "center", gap: "6px", padding: "2px 8px", borderRadius: "10px", background: ({ verified: isDark ? "#064e3b" : "#d1fae5", "in-progress": isDark ? "#1e3a8a" : "#dbeafe", unverified: isDark ? "#78350f" : "#fef3c7" })[verifyStatusOf(status)], color: ({ verified: isDark ? "#a7f3d0" : "#065f46", "in-progress": isDark ? "#bfdbfe" : "#1e40af", unverified: isDark ? "#fde68a" : "#92400e" })[verifyStatusOf(status)], fontSize: "11px", fontWeight: 600, whiteSpace: "nowrap" }), badgeDot: status => ({ width: "8px", height: "8px", borderRadius: "50%", background: ({ verified: "#10b981", "in-progress": "#3b82f6", unverified: "#f59e0b" })[verifyStatusOf(status)] }), iconButton: { padding: "4px 10px", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#374151", fontSize: "11px", fontWeight: 500, cursor: "pointer", display: "inline-flex", alignItems: "center", gap: "4px" }, iconRow: { display: "inline-flex", flexWrap: "wrap", gap: "6px" }, runModeWrap: { display: "inline-flex", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "10px", overflow: "hidden", fontSize: "11px", fontWeight: 600, userSelect: "none" }, runModeChip: active => ({ padding: "2px 10px", cursor: "pointer", background: active ? isDark ? "#1f2937" : "#fff" : "transparent", color: active ? isDark ? "#e5e7eb" : "#111827" : isDark ? "#9ca3af" : "#6b7280", borderRight: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}` }), runModeChipLast: active => ({ padding: "2px 10px", cursor: "pointer", background: active ? isDark ? "#1f2937" : "#fff" : "transparent", color: active ? isDark ? "#e5e7eb" : "#111827" : isDark ? "#9ca3af" : "#6b7280" }), headerLeft: { display: "inline-flex", flexWrap: "wrap", alignItems: "center", gap: "8px" }, modalBackdrop: { position: "fixed", inset: 0, background: "rgba(0,0,0,0.5)", display: "flex", alignItems: "center", justifyContent: "center", zIndex: 9999 }, modalBox: { background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#111827", borderRadius: "8px", padding: "20px", maxWidth: "720px", width: "92%", maxHeight: "85vh", overflowY: "auto", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, boxShadow: "0 10px 25px rgba(0,0,0,0.25)" }, modalHeader: { display: "flex", justifyContent: "space-between", alignItems: "center", marginBottom: "12px" }, modalTitle: { fontSize: "15px", fontWeight: 600 }, modalCloseBtn: { background: "transparent", border: "none", color: "inherit", fontSize: "20px", cursor: "pointer", padding: "0 6px", lineHeight: 1 }, formField: { display: "flex", flexDirection: "column", gap: "4px", marginBottom: "10px" }, formLabel: { fontSize: "12px", fontWeight: 500, color: isDark ? "#9ca3af" : "#4b5563" }, formInput: { padding: "6px 10px", fontSize: "13px", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#111827" : "#fff", color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, sectionHeading: { fontSize: "12px", fontWeight: 600, textTransform: "uppercase", letterSpacing: "0.04em", color: isDark ? "#9ca3af" : "#6b7280", margin: "12px 0 6px 0" }, primaryBtn: { padding: "6px 14px", background: "#D45D44", color: "white", border: "none", borderRadius: "4px", cursor: "pointer", fontSize: "13px", fontWeight: 500 }, benchCard: { padding: "8px 12px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", display: "flex", flexDirection: "column", gap: "8px" }, benchHeader: { display: "flex", flexWrap: "wrap", alignItems: "baseline", justifyContent: "space-between", gap: "6px 12px" }, benchTitle: { fontSize: "13px", fontWeight: 600, color: isDark ? "#e5e7eb" : "inherit" }, benchVersion: { fontSize: "11px", color: isDark ? "#9ca3af" : "#6b7280" }, benchHeaderRight: { display: "flex", flexWrap: "wrap", alignItems: "center", gap: "6px 10px", flexShrink: 0 }, benchChipRow: { display: "flex", alignItems: "center", gap: "6px", flexWrap: "wrap", margin: "2px 0 8px" }, benchChip: { padding: "2px 10px", fontSize: "12px", cursor: "pointer", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#374151", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, benchChipActive: { background: "#D45D44", color: "white", borderColor: "#D45D44" }, benchBlock: { border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderRadius: "4px", padding: "8px 10px", background: isDark ? "#111827" : "#fafafa" }, benchBlockTitle: { fontSize: "11px", fontWeight: 600, textTransform: "uppercase", letterSpacing: "0.04em", color: isDark ? "#9ca3af" : "#6b7280", marginBottom: "4px" }, benchWorkload: { fontSize: "11px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280", marginBottom: "6px", lineHeight: "1.3" }, benchRow: { display: "flex", justifyContent: "space-between", fontSize: "12px", padding: "2px 0" }, benchKey: { color: isDark ? "#9ca3af" : "#6b7280" }, benchVal: { color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontWeight: 500 }, benchNotes: { fontSize: "11px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280" }, benchLegend: { fontSize: "10px", fontStyle: "italic", color: isDark ? "#6b7280" : "#9ca3af", marginTop: "6px", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, benchEmpty: { fontSize: "12px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280" }, benchTableScroll: { overflowX: "auto", overflowY: "hidden", marginTop: "4px" }, benchTable: { display: "grid", columnGap: 0, rowGap: "3px", alignItems: "baseline", minWidth: "max-content" }, benchTableHead: { textAlign: "right", fontWeight: 500, fontSize: "11px", color: isDark ? "#9ca3af" : "#6b7280", paddingLeft: "16px", paddingBottom: "4px", whiteSpace: "nowrap" }, benchTableCornerHead: { paddingBottom: "4px", position: "sticky", left: 0, background: isDark ? "#111827" : "#fafafa", zIndex: 1, alignSelf: "stretch" }, benchTableSeparator: { gridColumn: "1 / -1", height: "1px", background: isDark ? "#374151" : "#e5e7eb", marginTop: "-3px" }, benchTableLabel: { textAlign: "left", fontSize: "12px", color: isDark ? "#9ca3af" : "#6b7280", whiteSpace: "nowrap", position: "sticky", left: 0, background: isDark ? "#111827" : "#fafafa", zIndex: 1 }, benchTableValue: { textAlign: "right", fontSize: "12px", color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontWeight: 500, paddingLeft: "16px", whiteSpace: "nowrap" }, benchTableValueMissing: { color: isDark ? "#6b7280" : "#9ca3af" } }); const VERIFY_LABEL = { verified: "Verified", "in-progress": "Final Verification In Progress", unverified: "Not Verified" }; const verifyStatusOf = v => typeof v === "string" ? VERIFY_LABEL[v] ? v : "unverified" : v ? "verified" : "unverified"; const cellVerifyStatus = (c, sel) => { if (!c) return "unverified"; const v = typeof c.verificationStatus === "function" ? c.verificationStatus(sel) : c.verificationStatus; return verifyStatusOf(v ?? c.verified); }; const LEGACY_MATCH_DIMS = [{ id: "variant", title: "Model Variant", optionsKey: "variants" }, { id: "quant", title: "Quantization", optionsKey: "quantizations" }, { id: "strategy", title: "Strategy", optionsKey: "strategies" }, { id: "nodes", title: "Nodes", optionsKey: "nodesOptions" }]; const matchDimSpecs = (config.matchDims || LEGACY_MATCH_DIMS).map(d => ({ ...d, options: d.options || config[d.optionsKey] || [] })); const overlayDimSpecs = config.overlayDims || []; const commandBuilder = config.commandBuilder || null; const DIMENSIONS = ["hw", ...matchDimSpecs.map(d => d.id)]; const optionVisible = (opt, sel) => typeof opt.showWhen !== "function" || opt.showWhen(sel); const optionDisabled = (opt, sel) => typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled; const visibleOptions = (spec, sel) => (spec.options || []).filter(o => optionVisible(o, sel)); const rowVisible = (spec, sel) => (typeof spec.showWhen !== "function" || spec.showWhen(sel)) && visibleOptions(spec, sel).length > 0; const overlayPick = sel => { const picked = []; for (const spec of config.overlayDims || []) { if (!rowVisible(spec, sel)) continue; const opt = (spec.options || []).find(o => o.id === sel[spec.id]); if (opt && !optionDisabled(opt, sel)) picked.push(opt); } return picked; }; const overlayPart = (sel, key) => { const out = []; for (const opt of overlayPick(sel)) { const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key]; if (add) out.push(...add); } return out; }; const overlayCompose = (cellFlags, sel) => { const strip = overlayPart(sel, "stripPrefixes"); const add = overlayPart(sel, "flags"); if (!strip.length) return [...cellFlags || [], ...add]; const used = new Set(); const replacementsFor = tok => { const out = []; add.forEach((f, i) => { if (used.has(i) || f.split(/[\s=]/)[0] !== tok) return; used.add(i); out.push(f); }); return out; }; const out = []; for (const f of cellFlags || []) { const tok = f.split(/[\s=]/)[0]; if (!strip.includes(tok)) out.push(f); else out.push(...replacementsFor(tok)); } add.forEach((f, i) => { if (!used.has(i)) out.push(f); }); return out; }; const optionSoft = (opt, sel) => typeof opt.soft === "function" ? opt.soft(sel) : !!opt.soft; const findCell = (cells, sel) => cells.find(c => DIMENSIONS.every(d => c.match[d] === sel[d])); const findBenchmark = (list, sel) => { const hits = (list || []).filter(b => Object.entries(b.match || ({})).every(([k, v]) => sel[k] === v)); return hits.sort((a, b) => Object.keys(b.match).length - Object.keys(a.match).length)[0] || null; }; const normalizeSpeed = speed => { if (!speed) return []; return Array.isArray(speed) ? speed : [speed]; }; const effectiveAccuracy = (entry, sel) => entry ? { ...config.defaultAccuracy && config.defaultAccuracy[sel.variant] || ({}), ...entry.accuracy || ({}) } : {}; const benchmarkIsEmpty = (entry, accuracy) => { for (const m of normalizeSpeed(entry && entry.speed)) { if (m && typeof m === "object") { for (const [key, v] of Object.entries(m)) { if (key === "workload") continue; if (v !== null && v !== undefined) return false; } } } if (accuracy && typeof accuracy === "object") { for (const v of Object.values(accuracy)) { if (v !== null && v !== undefined) return false; } } return true; }; const isOptionAvailable = (cells, sel, dim, value) => { const idx = DIMENSIONS.indexOf(dim); const higher = DIMENSIONS.slice(0, idx); return cells.some(c => c.match[dim] === value && higher.every(d => c.match[d] === sel[d])); }; const snapToValidCell = (cells, sel, dim, value) => { const idx = DIMENSIONS.indexOf(dim); const higher = DIMENSIONS.slice(0, idx); const lower = DIMENSIONS.slice(idx + 1); let best = null, bestLowerMatches = -1; for (const c of cells) { if (c.match[dim] !== value) continue; if (!higher.every(d => c.match[d] === sel[d])) continue; let s = 0; for (const d of lower) if (c.match[d] === sel[d]) s++; if (s > bestLowerMatches) { bestLowerMatches = s; best = c; } } if (!best) return sel; const next = { ...sel, [dim]: value }; for (const d of lower) next[d] = best.match[d]; return next; }; const validateSelection = (cells, parsed) => { const valid = {}; for (const dim of DIMENSIONS) { const want = parsed[dim]; const works = cells.some(c => c.match[dim] === want && DIMENSIONS.slice(0, DIMENSIONS.indexOf(dim)).every(d => c.match[d] === valid[d])); if (works) { valid[dim] = want; } else { const fallback = cells.find(c => DIMENSIONS.slice(0, DIMENSIONS.indexOf(dim)).every(d => c.match[d] === valid[d])); valid[dim] = fallback ? fallback.match[dim] : want; } } for (const spec of overlayDimSpecs) { const want = parsed[spec.id]; const opts = spec.options || []; const picked = opts.some(o => o.id === want) ? want : (spec.default ?? (opts[0] && opts[0].id)) ?? ""; const withPick = { ...valid, [spec.id]: picked }; const usable = visibleOptions(spec, withPick).filter(o => !optionDisabled(o, withPick)); valid[spec.id] = usable.some(o => o.id === picked) ? picked : (usable[0] && usable[0].id) ?? picked; } return valid; }; const resolveModelName = sel => { const keys = [`${sel.hw}|${sel.variant}|${sel.quant}`, `${sel.variant}|${sel.quant}`, `${sel.hw}|${sel.quant}`, sel.quant, sel.hw, "default"]; for (const k of keys) { const hit = config.modelNames[k]; if (hit) return hit; } return ""; }; const interpolate = (text, env, modelName) => text.replace(/{{(\w+)}}/g, (_, key) => key === "MODEL_NAME" ? modelName : env[key] ?? `{{${key}}}`); const parseNnodes = id => { if (Number.isInteger(id)) return id; if ((/^\d+$/).test(id || "")) return parseInt(id, 10); if (id === "single") return 1; const m = (/^multi-(\d+)$/).exec(id || ""); return m ? parseInt(m[1], 10) : 1; }; const cellNnodes = (cell, sel) => sel.nodes !== undefined ? parseNnodes(sel.nodes) : cell.nnodes || 1; const PD_SERVE_PORTS = { prefill: 30000, decode: 30100 }; const overlayEnv = sel => overlayPart(sel, "env"); const overlayHints = sel => overlayPart(sel, "hints"); const renderCommand = (cell, sel, envValues, mode = "python") => { if (!cell) return "# No command available for the current selection."; const modelName = resolveModelName(sel); const nnodes = cellNnodes(cell, sel); const multinode = nnodes > 1; const cellEnv = [...cell.env || [], ...overlayEnv(sel)]; const flags = overlayCompose(cell.flags, sel); if (multinode) { const PARALLELISM_ANCHORS = new Set(["--attn-dp-size", "--attention-data-parallel-size", "--dp-size", "--dp", "--tp-size", "--tp", "--sp-degree", "--ulysses-degree", "--ring-degree"]); let i = flags.reduce((last, flag, index) => PARALLELISM_ANCHORS.has(flag.split(/[\s=]/)[0]) ? index : last, -1); if (i === -1) i = flags.findIndex(f => f.startsWith("--model-path")); flags.splice(i + 1, 0, `--nnodes ${nnodes}`, `--node-rank {{NODE_RANK}}`, `--dist-init-addr {{NODE0_IP}}:20000`); } const pdServePort = PD_SERVE_PORTS[sel.pdMode]; if (pdServePort !== undefined) { for (let j = 0; j < flags.length; j++) { if (flags[j].split(/[\s=]/)[0] === "--port") { flags[j] = `--port ${pdServePort}`; } } } let cmd; if (mode === "docker") { const di = config.dockerImages || ({}); const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`] || di[`${sel.variant}|${sel.quant}`] || di[`${sel.hw}|${sel.quant}|${sel.strategy}`] || di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev"; const dockerRunCommand = typeof config.dockerRunCommand === "function" ? config.dockerRunCommand(sel) : config.dockerRunCommand || "sglang serve"; const portFlag = flags.find(x => x.split(/[\s=]/)[0] === "--port"); const servePort = portFlag ? portFlag.slice(("--port").length).trim() : "{{PORT}}"; const hostNetwork = multinode || typeof config.dockerHostNetworkWhen === "function" && config.dockerHostNetworkWhen(sel, { flags, env: cellEnv }); const vendorOf = hwId => { for (const [vendor, list] of Object.entries(HARDWARE_CATALOG)) { if (list.some(h => h.id === hwId)) return vendor; } const extra = (config.hardware || []).find(h => h.id === hwId); return extra && extra.vendor || "nvidia"; }; const catalogEntryOf = hwId => { const extra = (config.hardware || []).find(h => h.id === hwId); if (extra) return extra; for (const list of Object.values(HARDWARE_CATALOG)) { const hit = list.find(h => h.id === hwId); if (hit) return hit; } return null; }; const fabricFlagsOf = hwId => (catalogEntryOf(hwId) || ({})).multiNodeDockerFlags || []; const davinciLines = devices => { const lines = []; for (let i = 0; i < devices; i += 4) { const group = []; for (let k = i; k < Math.min(i + 4, devices); k++) { group.push(`--device=/dev/davinci${k}`); } lines.push(" " + group.join(" ")); } return lines; }; const gpuAccessLines = vendorOf(sel.hw) === "amd" ? ["docker run", " --device=/dev/kfd --device=/dev/dri", " --group-add video", " --cap-add=SYS_PTRACE --security-opt seccomp=unconfined", " --shm-size 32g"] : vendorOf(sel.hw) === "npu" ? ["docker run --privileged --shm-size=16g", ...davinciLines((catalogEntryOf(sel.hw) || ({})).npuDevices || 16), " --device=/dev/davinci_manager", " --device=/dev/hisi_hdc", " -v /usr/local/sbin:/usr/local/sbin", " -v /usr/local/Ascend/driver:/usr/local/Ascend/driver", " -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware", " -v /etc/ascend_install.info:/etc/ascend_install.info", " -v /var/queue_schedule:/var/queue_schedule", " -v ~/.cache/:/root/.cache/"] : ["docker run --gpus all", " --shm-size 32g"]; const dockerLines = [...gpuAccessLines, hostNetwork ? " --network host" : ` -p ${servePort}:${servePort}`, ...multinode ? fabricFlagsOf(sel.hw).map(f => " " + f) : [], ...vendorOf(sel.hw) === "npu" ? [] : [" -v ~/.cache/huggingface:/root/.cache/huggingface"], ...(config.dockerMounts || []).map(mount => ` -v ${mount}`), ...config.placeholders && config.placeholders.HF_TOKEN ? [` --env "HF_TOKEN={{HF_TOKEN}}"`] : [], ...cellEnv.map(e => ` --env ${e}`), " --ipc=host", ` ${image}`, ` ${dockerRunCommand}`, ...flags.map(f => " " + f)]; cmd = dockerLines.join(" \\\n"); } else { const flagBlock = flags.map(f => " " + f).join(" \\\n"); const envBlock = cellEnv.length ? cellEnv.join(" \\\n") + " \\\n" : ""; cmd = `${envBlock}sglang serve \\\n${flagBlock}`; } const hintLines = [...overlayHints(sel), ...multinode && config.multiNodeHints && config.multiNodeHints[sel.hw] ? config.multiNodeHints[sel.hw] : []]; if (hintLines.length) { const hint = hintLines.map(line => line.length ? "# " + line : "#").join("\n"); cmd = `${hint}\n${cmd}`; } cmd = interpolate(cmd, envValues, modelName); if (multinode) { const header = `# Multi-node (${nnodes} nodes). Run the same command on every node with:\n` + `# = 0 on the head node, 1..${nnodes - 1} on the others\n` + `# = IP of the head node (reachable from all others)`; cmd = `${header}\n${cmd}`; } return cmd; }; const ACCURACY_LABELS = config.accuracyLabels || []; const renderBenchmarkCard = entry => { const pct = entry && entry.latencyPercentile || config.latencyPercentile || "P50"; const SPEED_LABELS = [["ttft_ms", `TTFT (${pct})`, "ms"], ["tpot_ms", `TPOT (${pct})`, "ms"], ["tokens_per_sec_per_gpu", "throughput per gpu", "tok/s"], ["interactivity", "interactivity", "tokens/s/user", m => m.tpot_ms != null && m.tpot_ms !== 0 ? Math.round(1000 / m.tpot_ms * 10) / 10 : null]]; const WORKLOAD_KEYS = ["dataset", "isl", "osl", "max_concurrency"]; const fmt = (val, unit) => { if (val === null || val === undefined) return null; return `${val}${unit ? " " + unit : ""}`; }; const formatWorkloadParts = (workload, keys) => { if (!workload) return ""; const parts = []; if (keys.has("dataset") && workload.dataset) parts.push(workload.dataset); if (keys.has("isl") || keys.has("osl")) { if (workload.isl != null || workload.osl != null) { parts.push(`in/out=${workload.isl != null ? workload.isl : "?"}/${workload.osl != null ? workload.osl : "?"}`); } } if (keys.has("max_concurrency") && workload.max_concurrency != null) { parts.push(`max-concurrency=${workload.max_concurrency}`); } return parts.join(", "); }; const ALWAYS_PER_COLUMN = new Set(["max_concurrency"]); const ATOMIC_WORKLOAD_GROUPS = [["isl", "osl"]]; const partitionWorkload = measurements => { const shared = new Set(); const differing = new Set(); for (const k of WORKLOAD_KEYS) { const seen = new Set(); let anyPresent = false; for (const m of measurements) { const v = m && m.workload ? m.workload[k] : undefined; if (v != null) anyPresent = true; seen.add(v); } if (!anyPresent) continue; if (ALWAYS_PER_COLUMN.has(k) || seen.size > 1) differing.add(k); else shared.add(k); } for (const group of ATOMIC_WORKLOAD_GROUPS) { if (!group.some(k => differing.has(k))) continue; for (const k of group) if (shared.delete(k)) differing.add(k); } return { shared, differing }; }; const renderBenchTable = ({title, sharedText, colHeaders, rows, colCount, legend}) => { if (rows.length === 0) return null; const showColHeaders = colHeaders.length > 0 && colHeaders.some(h => h !== ""); return
{title}
{sharedText &&
{sharedText}
} {}
1 ? { tabIndex: 0, role: "group", "aria-label": sharedText ? `${title} — ${sharedText}` : title } : {}}>
{showColHeaders &&
} {showColHeaders && colHeaders.map((h, i) =>
{h}
)} {showColHeaders &&
} {rows.map(r => [
{r.label}
, ...r.values.map((v, i) =>
{v !== null ? v : "—"}
)])}
{legend &&
{(Array.isArray(legend) ? legend : [legend]).map((line, i) =>
{line}
)}
}
; }; const buildSpeedTable = measurements => { if (measurements.length === 0) return null; const {shared, differing} = partitionWorkload(measurements); const sharedText = formatWorkloadParts(measurements[0] && measurements[0].workload, shared); const colHeaders = measurements.map(m => formatWorkloadParts(m && m.workload, differing)); const rows = SPEED_LABELS.map(tup => { const [key, label, unit, compute] = tup; const values = measurements.map(m => { const raw = compute ? compute(m) : m[key]; return fmt(raw, unit); }); return { label, values }; }); return { title: "Speed", sharedText, colHeaders, rows, colCount: measurements.length, legend: [`throughput per gpu = (input+output tokens)/elapsed/GPU`, `interactivity = 1000/TPOT(ms) (tokens/s/user)`] }; }; const buildAccuracyTable = accuracy => { if (!accuracy) return null; const rows = ACCURACY_LABELS.map(([key, label, unit]) => { const v = fmt(accuracy[key], unit); if (v === null) return null; return { label, values: [v] }; }).filter(r => r !== null); if (rows.length === 0) return null; return { title: "Accuracy", sharedText: null, colHeaders: [], rows, colCount: 1 }; }; const accuracy = effectiveAccuracy(entry, sel); const isEmpty = benchmarkIsEmpty(entry, accuracy); const measurements = !isEmpty ? normalizeSpeed(entry && entry.speed) : []; const accuracyTable = !isEmpty ? buildAccuracyTable(accuracy) : null; const speedTable = !isEmpty ? buildSpeedTable(measurements) : null; const hasBenchCmds = !isEmpty && buildBenchCommands(entry, sel) !== null; return
Benchmark
{!isEmpty && entry && entry.sglang_version &&
measured on sglang {entry.sglang_version}
} {hasBenchCmds && }
{isEmpty ?
Benchmark data pending for this combination — submit yours via the Playground's Submit ↗ button.
: <> {accuracyTable && renderBenchTable(accuracyTable)} {speedTable && renderBenchTable(speedTable)} {entry && entry.notes &&
{entry.notes}
} }
; }; const buildBenchCommands = (entry, sel) => { const bc = config.benchmarkCommands; if (!bc) return null; const acc = effectiveAccuracy(entry, sel); const accuracy = []; if (bc.accuracy) { for (const [key, label] of ACCURACY_LABELS) { if (acc[key] == null) continue; const tmpl = bc.accuracy[key]; const resolved = typeof tmpl === "string" ? tmpl : tmpl && tmpl[sel.variant] || null; if (resolved) accuracy.push({ key, label, template: resolved }); } } let speed = null; if (bc.speed && entry) { const ms = normalizeSpeed(entry.speed).filter(m => m && m.workload && m.workload.max_concurrency != null); const concurrencies = [...new Set(ms.map(m => m.workload.max_concurrency))].sort((a, b) => a - b); if (concurrencies.length) { speed = { template: bc.speed, concurrencies, workload: ms[0].workload, numPromptsOf: c => { const m = ms.find(x => x.workload.max_concurrency === c); if (m && m.workload.num_prompts != null) return m.workload.num_prompts; const tbl = bc.numPromptsByConc; if (tbl && tbl[c] != null) return tbl[c]; return Math.max(c * 2, 200); } }; } } if (accuracy.length === 0 && !speed) return null; return { accuracy, speed }; }; const buildHardwareGroups = () => { const supported = new Set(config.supportedHardware); const catalog = {}; for (const [vendor, list] of Object.entries(HARDWARE_CATALOG)) catalog[vendor] = [...list]; for (const hw of config.hardware || []) { const vendor = hw.vendor || "nvidia"; const list = catalog[vendor] || (catalog[vendor] = []); const entry = { id: hw.id, label: hw.label, vram: hw.vram }; const i = list.findIndex(x => x.id === hw.id); if (i >= 0) list[i] = entry; else list.push(entry); } const groups = []; for (const [vendor, list] of Object.entries(catalog)) { const items = list.filter(hw => supported.has(hw.id)).map(hw => ({ id: hw.id, label: hw.label, subtitle: hw.vram })); if (items.length) groups.push({ label: vendor.toUpperCase(), items }); } if (config.groupHardware === false) { return [{ label: null, items: groups.flatMap(group => group.items) }]; } return groups; }; const initialSelectionFromCells = () => { const first = (config.cells || [])[0]; const sel = Object.fromEntries(DIMENSIONS.map(d => [d, first ? first.match[d] : ""])); for (const spec of overlayDimSpecs) { const opts = spec.options || []; sel[spec.id] = (spec.default ?? (opts[0] && opts[0].id)) ?? ""; } if (!commandBuilder) return sel; return { ...sel, hw: commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || "", ...commandBuilder.defaultSelection || ({}) }; }; const normalizeBuilderSelection = parsed => { const out = { ...initialSelectionFromCells(), ...parsed }; if (!(config.supportedHardware || []).includes(out.hw)) { out.hw = commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || ""; } for (const spec of overlayDimSpecs) { if (spec.kind === "number") { const value = Number.parseInt(out[spec.id], 10); out[spec.id] = Math.min(spec.max, Math.max(spec.min, Number.isFinite(value) ? value : Number(spec.default ?? spec.min))); continue; } const options = spec.options || []; if (!options.some(option => option.id === out[spec.id])) { out[spec.id] = (spec.default ?? options[0]?.id) ?? ""; } } for (const [key, bounds] of Object.entries(commandBuilder.resource?.limits || ({}))) { const fallback = Number((commandBuilder.defaultSelection?.[key] ?? bounds.min) ?? 1); const value = Number.parseInt(out[key], 10); out[key] = Math.min(bounds.max, Math.max(bounds.min, Number.isFinite(value) ? value : fallback)); } for (const key of ["tp_size", "ulysses_degree", "ring_degree"]) { const value = Number.parseInt(out[key], 10); out[key] = Number.isFinite(value) && value > 0 ? value : 1; } out.topology_mode = out.topology_mode === "manual" ? "manual" : "auto"; return out; }; const placeholderDefaults = schema => { const out = {}; for (const [k, v] of Object.entries(schema || ({}))) out[k] = v.default ?? ""; return out; }; const [isDark, setIsDark] = useState(false); useEffect(() => { const check = () => { const html = document.documentElement; setIsDark(html.classList.contains("dark") || html.getAttribute("data-theme") === "dark" || html.style.colorScheme === "dark"); }; check(); const observer = new MutationObserver(check); observer.observe(document.documentElement, { attributes: true, attributeFilter: ["class", "data-theme", "style"] }); return () => observer.disconnect(); }, []); const STORAGE_KEY = "sglang-deploy-env"; const [env, setEnv] = useState(() => placeholderDefaults(config.placeholders)); useEffect(() => { try { const raw = window.localStorage.getItem(STORAGE_KEY); if (raw) { const parsed = JSON.parse(raw); setEnv({ ...placeholderDefaults(config.placeholders), ...parsed }); } } catch {} }, []); const saveEnv = next => { setEnv(next); try { window.localStorage.setItem(STORAGE_KEY, JSON.stringify(next)); } catch {} }; const [sel, setSel] = useState(() => initialSelectionFromCells()); const [selectionHydrated, setSelectionHydrated] = useState(false); const INTERNAL_HASH_STATE_KEY = "__sglangDeployInternalHash"; const DEPLOYMENT_COMPONENT_ID = "deployment-configurator"; useEffect(() => { const hydrate = () => { const raw = window.location.hash.replace(/^#/, ""); if (!raw) return; const params = new URLSearchParams(raw); const initial = initialSelectionFromCells(); const parsed = { ...initial }; let touched = false; params.forEach((value, key) => { if ((key in parsed)) { parsed[key] = value; touched = true; } }); if (!touched) return; setSel(commandBuilder ? normalizeBuilderSelection(parsed) : validateSelection(config.cells, parsed)); const historyState = window.history.state; const isInternalHash = historyState && typeof historyState === "object" && historyState[INTERNAL_HASH_STATE_KEY] === `#${raw}`; if (isInternalHash) return; const el = document.getElementById(DEPLOYMENT_COMPONENT_ID); if (el) el.scrollIntoView({ behavior: "smooth", block: "start" }); }; hydrate(); setSelectionHydrated(true); window.addEventListener("hashchange", hydrate); return () => window.removeEventListener("hashchange", hydrate); }, []); useEffect(() => { if (!selectionHydrated) return; const target = "#" + new URLSearchParams(sel).toString(); if (window.location.hash !== target) { const historyState = window.history.state && typeof window.history.state === "object" ? window.history.state : {}; window.history.replaceState({ ...historyState, [INTERNAL_HASH_STATE_KEY]: target }, "", target); } window.dispatchEvent(new CustomEvent("sglang-deploy-sel", { detail: sel })); }, [sel, selectionHydrated]); const [modal, setModal] = useState(null); useEffect(() => { if (modal === null) return; const onKey = e => { if (e.key === "Escape") setModal(null); }; const prev = document.body.style.overflow; document.body.style.overflow = "hidden"; window.addEventListener("keydown", onKey); return () => { window.removeEventListener("keydown", onKey); document.body.style.overflow = prev; }; }, [modal]); const [copied, setCopied] = useState(false); const [curlCopied, setCurlCopied] = useState(false); const [envDraft, setEnvDraft] = useState(env); const [benchConc, setBenchConc] = useState(null); const [benchAcc, setBenchAcc] = useState(null); const [benchCopied, setBenchCopied] = useState(null); const configuredRunModes = typeof config.runModes === "function" ? config.runModes(sel) : config.runModes; const runModes = configuredRunModes || ["python", "docker"]; const [runMode, setRunMode] = useState(runModes[0]); const [builderScope, setBuilderScope] = useState("base"); const [builderServerSetting, setBuilderServerSetting] = useState(null); const [builderAdvanced, setBuilderAdvanced] = useState(false); const [serveExpanded, setServeExpanded] = useState(false); const [requestExpanded, setRequestExpanded] = useState(false); const [builderHeadAddress, setBuilderHeadAddress] = useState(""); const [builderNodeRank, setBuilderNodeRank] = useState(0); const [blockedNote, setBlockedNote] = useState(null); const flashBlockedNote = (dim, reason) => { const note = { dim, reason }; setBlockedNote(note); setTimeout(() => setBlockedNote(cur => cur === note ? null : cur), 4000); }; useEffect(() => { if (builderNodeRank >= Number(sel.nodes || 1)) setBuilderNodeRank(0); }, [sel.nodes, builderNodeRank]); const hasRunMode = runModes.includes(runMode); const fallbackRunMode = runModes[0]; const activeRunMode = hasRunMode ? runMode : fallbackRunMode; useEffect(() => { if (!hasRunMode) setRunMode(fallbackRunMode); }, [hasRunMode, fallbackRunMode]); useEffect(() => { if (modal === "env") setEnvDraft(env); }, [modal, env]); const [mambaRatio, setMambaRatio] = useState(null); useEffect(() => { const onRatio = e => setMambaRatio(e.detail && (e.detail.baseRatio || e.detail.ratio) || null); window.addEventListener("sglang-k3-mamba-ratio", onRatio); return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio); }, []); const s = makeStyles(isDark); const cell = commandBuilder ? commandBuilder.resolveDeployment(sel) : findCell(config.cells, sel); const builderMeta = cell && cell.builder || ({}); const verifyStatus = cellVerifyStatus(cell, sel); const cellWithRatio = (() => { if (!cell || !mambaRatio) return cell; if (cell.flags.some(f => f.startsWith("--mamba-full-memory-ratio") || f.startsWith("--max-mamba-cache-size"))) return cell; const flags = [...cell.flags]; const line = `--mamba-full-memory-ratio ${mambaRatio}`; const i = flags.findIndex(f => f.startsWith("--host")); if (i >= 0) flags.splice(i, 0, line); else flags.push(line); return { ...cell, flags }; })(); const commandEnv = commandBuilder ? { ...env, NODE_RANK: String(builderNodeRank), NODE0_IP: builderHeadAddress || "" } : env; const command = renderCommand(cellWithRatio, sel, commandEnv, activeRunMode); const effFlags = cell ? overlayCompose(cell.flags, sel) : []; const specAlgoFlag = effFlags.find(f => f.split(/[\s=]/)[0] === "--speculative-algorithm"); const specMrrFlag = effFlags.find(f => f.split(/[\s=]/)[0] === "--max-running-requests"); const mtpHint = !!specAlgoFlag && !specMrrFlag; const specPinnedHint = !!specAlgoFlag && !!specMrrFlag; const specMrrValue = specMrrFlag ? specMrrFlag.split(/[\s=]/).filter(Boolean)[1] || "" : ""; const SPEC_ALGO_LABEL = { EAGLE: "MTP", EAGLE3: "MTP", FROZEN_KV_MTP: "MTP", DSPARK: "DSpark", DFLASH: "DFlash", NGRAM: "N-gram", STANDALONE: "standalone draft" }; const specAlgoName = (() => { if (!specAlgoFlag) return "MTP"; const v = specAlgoFlag.split(/[\s=]/).filter(Boolean)[1] || ""; return SPEC_ALGO_LABEL[v.toUpperCase()] || v || "MTP"; })(); const renderWarn = text => { const out = []; const re = /\[([^\]]+)\]\(#([^)]+)\)/g; let last = 0; for (let m; m = re.exec(text); last = m.index + m[0].length) { if (m.index > last) out.push(text.slice(last, m.index)); const anchor = m[2]; out.push(); } if (last < text.length) out.push(text.slice(last)); return out; }; const modelName = resolveModelName(sel); const curlTemplate = typeof config.curl === "function" ? config.curl(sel, cell) : config.curl; const curlText = interpolate(curlTemplate || "", env, modelName); const hwGroups = buildHardwareGroups(); const benchEntry = benchmarks ? findBenchmark(benchmarks, sel) : null; const isOverlayDim = dim => overlayDimSpecs.some(d => d.id === dim); const findOption = (dim, value) => { const spec = [...matchDimSpecs, ...overlayDimSpecs].find(d => d.id === dim); return spec && (spec.options || []).find(o => o.id === value); }; const isEnabled = (dim, value) => { const opt = findOption(dim, value); if (opt && optionDisabled(opt, sel)) return false; if (commandBuilder && dim === "hw") return true; return isOverlayDim(dim) || isOptionAvailable(config.cells || [], sel, dim, value); }; const reseatHiddenPicks = next => { let out = next; for (const spec of [...matchDimSpecs, ...overlayDimSpecs]) { const opts = visibleOptions(spec, out).filter(o => !optionDisabled(o, out)); if (!opts.length) continue; if (!opts.some(o => o.id === out[spec.id])) { out = { ...out, [spec.id]: opts[0].id }; } } return out; }; const recommendedBuilderRecipe = hw => { const recipes = commandBuilder.resource?.verifiedRecipes || []; return recipes.find(entry => entry.hw === hw && entry.default) || recipes.find(entry => entry.hw === hw); }; const handleSelect = (dim, value) => { if (commandBuilder) { setSel(prev => { let next = { ...prev, [dim]: value }; if (dim === "hw") { const currentRecipe = recommendedBuilderRecipe(prev.hw); const nextRecipe = recommendedBuilderRecipe(value); const resourcesFollowPlatformDefault = !!currentRecipe && Number(prev.nodes) === Number(currentRecipe.nodes) && Number(prev.gpus_per_node) === Number(currentRecipe.gpus_per_node); next = { ...next, nodes: resourcesFollowPlatformDefault ? nextRecipe?.nodes ?? next.nodes : next.nodes, gpus_per_node: resourcesFollowPlatformDefault ? nextRecipe?.gpus_per_node ?? next.gpus_per_node : next.gpus_per_node, topology_mode: "auto", tp_size: resourcesFollowPlatformDefault ? nextRecipe?.tp_size ?? 1 : next.tp_size, ulysses_degree: resourcesFollowPlatformDefault ? nextRecipe?.ulysses_degree ?? 1 : next.ulysses_degree, ring_degree: resourcesFollowPlatformDefault ? nextRecipe?.ring_degree ?? 1 : next.ring_degree, placement: resourcesFollowPlatformDefault ? nextRecipe?.placement || "auto" : next.placement, encoder: resourcesFollowPlatformDefault ? nextRecipe?.encoder || "auto" : next.encoder }; } return reseatHiddenPicks(normalizeBuilderSelection(next)); }); return; } setSel(prev => reseatHiddenPicks(isOverlayDim(dim) ? { ...prev, [dim]: value } : snapToValidCell(config.cells, prev, dim, value))); }; const commitBuilderNumber = (event, currentValue, bounds, commit) => { const parsed = Number(event.currentTarget.value); if (!Number.isInteger(parsed)) { event.currentTarget.value = String(currentValue); return; } const value = Math.min(bounds.max, Math.max(bounds.min, parsed)); event.currentTarget.value = String(value); commit(value); }; const renderBuilderNumberInput = ({identity, value, min, max, label, onCommit}) => event.currentTarget.select()} onBlur={event => commitBuilderNumber(event, value, { min, max }, onCommit)} onKeyDown={event => { if (event.key === "Enter") event.currentTarget.blur(); }} />; const updateBuilderResource = (key, delta) => { if (!commandBuilder) return; const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); setSel(prev => { const value = Math.min(bounds.max, Math.max(bounds.min, Number(prev[key]) + delta)); return normalizeBuilderSelection({ ...prev, [key]: value, topology_mode: "auto" }); }); }; const setBuilderResource = (key, rawValue) => { if (!commandBuilder) return; const value = Number.parseInt(rawValue, 10); if (!Number.isFinite(value)) return; const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); setSel(prev => normalizeBuilderSelection({ ...prev, [key]: Math.min(bounds.max, Math.max(bounds.min, value)), topology_mode: "auto" })); }; const editBuilderTopology = (key, value) => { if (!commandBuilder) return; setSel(prev => normalizeBuilderSelection({ ...prev, topology_mode: "manual", [key]: Number.parseInt(value, 10) || 1 })); }; const handleCopy = () => { navigator.clipboard.writeText(command); setCopied(true); setTimeout(() => setCopied(false), 1200); }; const copyCurl = () => { navigator.clipboard.writeText(curlText); setCurlCopied(true); setTimeout(() => setCurlCopied(false), 1200); }; const copyBench = (key, text) => { navigator.clipboard.writeText(text); setBenchCopied(key); setTimeout(() => setBenchCopied(null), 1200); }; const placeholderGroups = (() => { const out = { command: [], curl: [] }; for (const [key, meta] of Object.entries(config.placeholders || ({}))) { (out[meta.target] || (out[meta.target] = [])).push({ key, ...meta }); } return out; })(); const renderButton = (item, dim, selectedId) => { const checked = selectedId === item.id; const disabled = !isEnabled(dim, item.id); return ; }; const renderFlatSection = (title, options, dim, selectedId) =>
{title}
{options.map(item => renderButton(item, dim, selectedId))}
; const maxHwCols = Math.max(...hwGroups.map(x => x.items.length)); if (commandBuilder) { const scopeLabel = { base: "Setup", serve: "Server", request: "Request" }; const scopedDims = scope => overlayDimSpecs.filter(dim => { if ((dim.scope || "base") !== scope) return false; if (dim.kind === "number") { return typeof dim.showWhen !== "function" || dim.showWhen(sel); } return rowVisible(dim, sel); }); const baseDims = scopedDims("base"); const serveDims = scopedDims("serve"); const requestDims = scopedDims("request"); const errors = builderMeta.errors || []; const warnings = builderMeta.warnings || []; const invalid = errors.length > 0; const totalGpus = Number(sel.nodes) * Number(sel.gpus_per_node); const topology = builderMeta.topology || ({}); const verification = builderMeta.verification || ({}); const scopeIsVerified = scope => scopedDims(scope).every(dim => { const option = (dim.options || []).find(entry => entry.id === sel[dim.id]); if (option && optionSoft(option, sel)) return false; const predicate = option?.verifiedWhen ?? dim.verifiedWhen; return typeof predicate === "function" ? !!predicate(sel) : predicate !== false; }); const serveStatus = invalid ? "error" : scopeIsVerified("serve") ? verification.serve || verifyStatus : "unverified"; const requestStatus = invalid ? "error" : scopeIsVerified("request") ? verification.request || verifyStatus : "unverified"; const statusText = status => ({ verified: "Verified", unverified: "Unverified", "in-progress": "Verification in progress", error: "Invalid configuration" })[status] || "Unverified"; const activeServerSetting = serveDims.find(dim => dim.id === builderServerSetting) || serveDims[0]; const selectedOption = dim => (dim.options || []).find(option => option.id === sel[dim.id]); const effectiveSetting = dim => builderMeta.resolvedSettings?.[dim.id] || selectedOption(dim)?.label || sel[dim.id] || "—"; const recommendedRecipe = recommendedBuilderRecipe(sel.hw); const recommendedInUse = !!recommendedRecipe && Number(sel.nodes) === recommendedRecipe.nodes && Number(sel.gpus_per_node) === recommendedRecipe.gpus_per_node && sel.topology_mode === "auto" && serveDims.every(dim => { const expected = recommendedRecipe[dim.id] ?? dim.default; if (dim.id === "attention") return sel.attention === "platform"; return expected === undefined || sel[dim.id] === expected || ["placement", "encoder"].includes(dim.id) && sel[dim.id] === "auto"; }); const restoreRecommendedRecipe = () => { if (!recommendedRecipe) return; setSel(prev => reseatHiddenPicks(normalizeBuilderSelection({ ...prev, ...Object.fromEntries(serveDims.map(dim => [dim.id, (recommendedRecipe[dim.id] ?? dim.default) ?? dim.options?.[0]?.id])), nodes: recommendedRecipe.nodes, gpus_per_node: recommendedRecipe.gpus_per_node, topology_mode: "auto", tp_size: recommendedRecipe.tp_size, ulysses_degree: recommendedRecipe.ulysses_degree, ring_degree: recommendedRecipe.ring_degree, placement: recommendedRecipe.placement || "auto", attention: "platform", precision: "native", encoder: recommendedRecipe.encoder || "auto", execution: recommendedRecipe.execution || "eager" }))); }; const renderBuilderChoice = (item, dim) => { const checked = sel[dim.id] === item.id; const disabled = !isEnabled(dim.id, item.id); const soft = !disabled && optionSoft(item, sel); const reason = disabled ? item.disableReason || "Not available for this configuration" : soft ? item.softReason || "Runs, but this combination is not a verified recipe yet." : ""; return ; }; const renderBuilderDimension = dim =>
{dim.title} {dim.description && {dim.description}}
5 ? "compact" : "normal"}> {visibleOptions(dim, sel).map(option => renderBuilderChoice(option, dim))}
{blockedNote && blockedNote.dim === dim.id &&

{blockedNote.reason}

}
; const renderStepper = (key, label, detail) => { const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); return
{label} {detail && {detail}}
{renderBuilderNumberInput({ identity: `${key}-${sel[key]}`, value: sel[key], min: bounds.min, max: bounds.max, label, onCommit: value => setBuilderResource(key, value) })}
; }; const renderBaseScope = () =>
{recommendedRecipe &&
{} {recommendedRecipe.unverified ? "Derived recipe" : "Verified recipe"} · {sel.hw.toUpperCase()} {[`${recommendedRecipe.nodes * recommendedRecipe.gpus_per_node} GPUs`, recommendedRecipe.tp_size > 1 && `TP ${recommendedRecipe.tp_size}`, `Ulysses ${recommendedRecipe.ulysses_degree}`, recommendedRecipe.ring_degree > 1 && `Ring ${recommendedRecipe.ring_degree}`, ({ resident: "Resident", fsdp: "FSDP", offload: "Layerwise offload" })[recommendedRecipe.placement]].filter(Boolean).join(" · ")}
{renderStatus(recommendedRecipe.unverified ? "unverified" : "verified")} {recommendedInUse ? In use : }
}
Hardware
{hwGroups.flatMap(group => group.items).map(item => { const selected = sel.hw === item.id; return ; })}
{}
Resources {builderMeta.topologySummary || "No valid topology"}
{renderStepper("nodes", "Nodes")} {renderStepper("gpus_per_node", "GPUs / node")}
{Number(sel.nodes) > 1 &&

{sel.nodes} nodes × {sel.gpus_per_node} {sel.hw.toUpperCase()} = {totalGpus} GPUs

} {builderAdvanced &&

Auto uses an exact verified recipe when one exists; manual values are allowed when the model constraints remain valid.

{[["tp_size", "Tensor parallel", [1, 2, 4, 8]], ["ulysses_degree", "Ulysses", [1, 2, 4, 8, 16]], ["ring_degree", "Ring", [1, 2, 4, 8]]].map(([key, label, values]) => )}
} {(errors.length > 0 || warnings.length > 0) &&
{(errors.length ? errors : warnings).map((message, index) =>

{message}

)}
}
{baseDims.map(renderBuilderDimension)}
; const renderSettingEditor = (dim, className = "", direct = false) => { if (!dim) return null; const options = visibleOptions(dim, sel); const currentOption = selectedOption(dim); return
{direct ? dim.title : `${dim.title} options`} {dim.description &&

{dim.description}

}
{dim.quality && {dim.quality}}
{dim.kind === "number" ?
{renderBuilderNumberInput({ identity: `${dim.id}-${sel[dim.id]}`, value: sel[dim.id], min: dim.min, max: dim.max, label: dim.title, onCommit: value => setSel(prev => ({ ...prev, [dim.id]: value })) })} {dim.unit || "outputs"}
:
{options.map(option => renderBuilderChoice(option, dim))}
} {blockedNote && blockedNote.dim === dim.id &&

{blockedNote.reason}

} {(currentOption?.description || dim.learnMore) &&
{currentOption?.description &&

{currentOption.description}

} {} {dim.learnMore && Learn more } {dim.docsHref && SGLang docs }
}
; }; const renderServerScope = () =>
{serveDims.map(dim => { const isActive = dim.id === activeServerSetting?.id; const option = selectedOption(dim); const recommended = typeof option?.recommendedWhen === "function" ? option.recommendedWhen(sel) : !!option?.recommended; return
{isActive && renderSettingEditor(dim, "sgd-builder-context--inline")}
; })}
{renderSettingEditor(activeServerSetting, "sgd-builder-context--rail")}
; const renderRequestScope = () =>
{requestDims.map(dim =>
{renderSettingEditor(dim, "", true)}
)}
; const renderScopeControls = () => { if (builderScope === "base") return renderBaseScope(); if (builderScope === "serve") return renderServerScope(); return renderRequestScope(); }; const renderStatus = status => ; const renderOutputCard = type => { const serve = type === "serve"; const text = serve ? command : curlText; const canExpand = text.split("\n").length > 9; const expanded = serve ? serveExpanded : requestExpanded; const setExpanded = serve ? setServeExpanded : setRequestExpanded; const status = serve ? serveStatus : requestStatus; const emphasized = builderScope === "base" || builderScope === type; return
{serve ? "1" : "2"}
{serve ? "Serve" : "Request"} {serve ? `${sel.hw.toUpperCase()} · ${activeRunMode === "docker" ? "Docker" : "Python"}` : "cURL"}
{renderStatus(status)}
{serve && runModes.length > 1 &&
{runModes.map(mode => )}
} {serve && Number(sel.nodes) > 1 &&
} 
{text}
{invalid &&
{errors[0]}
}
{canExpand && }
; }; return
{renderScopeControls()}
{builderScope !== "request" && renderOutputCard("serve")} {builderScope !== "serve" && renderOutputCard("request")}
{modal === "env" &&
setModal(null)}>
event.stopPropagation()}>
Command variables
{["command", "curl"].map(target => placeholderGroups[target].length > 0 &&
{target === "command" ? "Serve" : "Request"}
{placeholderGroups[target].map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: event.target.value })} />
)}
)}
}
; } return
{}
Hardware Platform
{hwGroups.map(g =>
{g.label &&
{g.label}
}
{g.items.map(item => renderButton(item, "hw", sel.hw))} {Array.from({ length: maxHwCols - g.items.length }).map((_, i) =>
)}
)}
{matchDimSpecs.filter(d => rowVisible(d, sel)).map(d =>
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
)} {overlayDimSpecs.filter(d => rowVisible(d, sel)).map(d =>
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
)} {}
Command:
{cell && cell.redirect ? cell.warn &&
⚠️ {renderWarn(cell.warn)}
: <>
{VERIFY_LABEL[verifyStatus]}
{runModes.map((mode, index) => setRunMode(mode)} onKeyDown={e => { if (e.key !== "Enter" && e.key !== " ") return; e.preventDefault(); setRunMode(mode); }} role="tab" tabIndex={0} aria-selected={activeRunMode === mode}> {mode === "docker" ? "Docker" : "Python"} )} 
{command}
{cell && cell.warn &&
⚠️ {renderWarn(cell.warn)}
} {mtpHint &&
⚠️ Speculative decoding ({specAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency.
} {specPinnedHint &&
ℹ️ Speculative decoding ({specAlgoName}) is on and this recipe pins --max-running-requests to {specMrrValue}. Adjust it to match your target concurrency — if you remove the flag, SGLang falls back to 48.
} }
{} {benchmarks && cell && renderBenchmarkCard(benchEntry)} {} {config.showPlaygroundLink !== false &&
Need to go beyond the verified matrix?
} {} {modal === "curl" &&
setModal(null)}>
e.stopPropagation()}>
cURL example
Model: {modelName || "(unresolved)"} 
{curlText}

Edit CURL_HOST / CURL_PORT in the Env panel.

} {} {modal === "env" &&
setModal(null)}>
e.stopPropagation()}>
Env / placeholder values
{placeholderGroups.curl.length > 0 &&
cURL placeholders
{placeholderGroups.curl.map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: e.target.value })} />
)}
} {placeholderGroups.command.length > 0 &&
Command placeholders
{placeholderGroups.command.map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: e.target.value })} />
)}
}

Values persist in localStorage and are reused the next time you visit any cookbook.

} {} {modal === "bench" && benchEntry && (() => { const bc = buildBenchCommands(benchEntry, sel); if (!bc) return null; const selSummary = [sel.hw && sel.hw.toUpperCase(), sel.variant, sel.quant && sel.quant.toUpperCase(), sel.strategy, sel.kvDsaPair, sel.nodes].filter(part => part !== undefined && part !== null && part !== "").join(" · "); let selConc = null; let speedCmd = null; if (bc.speed) { selConc = bc.speed.concurrencies.includes(benchConc) ? benchConc : bc.speed.concurrencies[0]; const w = bc.speed.workload; speedCmd = interpolate(bc.speed.template, { ...env, DATASET: w.dataset, ISL: w.isl, OSL: w.osl, MAX_CONCURRENCY: selConc, NUM_PROMPTS: bc.speed.numPromptsOf(selConc) }, modelName); } let selAcc = null; let accCmd = null; if (bc.accuracy.length > 0) { selAcc = bc.accuracy.find(a => a.key === benchAcc) || bc.accuracy[0]; accCmd = interpolate(selAcc.template, env, modelName); } return
setModal(null)}>
e.stopPropagation()}>
Benchmark commands

For {selSummary}. Start the server with the Deploy command above, then run these against it.

{selAcc &&
Accuracy
{bc.accuracy.length > 1 &&
benchmark: {bc.accuracy.map(a => )}
}
{selAcc.label}
{accCmd}
{bc.accuracy.length > 1 &&

Switch the benchmark chip to see each eval's command.

}
} {bc.speed &&
Speed
{bc.speed.concurrencies.length > 1 &&
max-concurrency: {bc.speed.concurrencies.map(c => )}
}
max-concurrency = {selConc}
{speedCmd}

One command — switch the concurrency chip (or edit --max-concurrency) to reproduce each Speed column.

}

Edit CURL_HOST / CURL_PORT in the Env panel.

; })()}
; }; export const DiffusionModelTags = ({tags = []}) => { const normalizedTags = Array.isArray(tags) ? tags : [tags]; return
{normalizedTags.map(tag => {tag} )}
; }; ## 1. Quick start Use Linux with NVIDIA CUDA. This model is not yet in a released package; use a checkout containing this implementation, not an unmodified release image. Your Hugging Face account must have access to the checkpoint; authenticate with `hf auth login` or supply `HF_TOKEN` in the server environment. Install from the repo root: ```bash theme={null} uv pip install -e "python[diffusion]" --prerelease=allow ``` Start with Pro-distill and resident weights on B200, GB300, or RTX PRO 6000 (96 GB). The picker separates verified lightweight HTTP requests from full-size offline generation and unverified custom topologies. See the [installation guide](/docs/sglang-diffusion/installation) for platform setup. The request returns a video job ID, not the video bytes. Poll `GET /v1/videos/{id}` until `status` is `completed`, then download `GET /v1/videos/{id}/content` for the MP4 containing both video and audio. Treat `failed` as an error; delete the job with `DELETE /v1/videos/{id}` when finished. See [Download a completed video](#5-download-a-completed-video) for a complete polling example. ## 2. Model capabilities Kandinsky 6.0 generates video and audio jointly from a text prompt, optionally conditioned on a reference image. One DiT predicts both modalities at each denoising step. A Reason1 encoder derived from Qwen2.5-VL supplies token features, and CLIP supplies pooled text features. Both request modes use the same server; image conditioning does not require switching checkpoints. Choose Pro-distill for the 10-step path, or Pro for 50-step classifier-free guidance. The recipes cover 768 x 512, 121-frame clips at 24 fps. They use native SGLang components, without calling a third-party model pipeline. Validation compares against the initial SGLang implementation; it does not establish parity with the unpublished Kandinsky6 Diffusers implementation. | Checkpoint | Scheduler | Default steps | Default guidance | | - | - | -: | -: | | `kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers` | PiFlow | 10 | 1.0 | | `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | Flow-match Euler | 50 | 5.0 | Pro-distill only accepts guidance 1.0 and does not encode a negative prompt. The scheduler is loaded from the checkpoint. For a local directory with an unrelated name, specify `--model-id Kandinsky-6.0-Pro-distill-5s-Diffusers` to select its sampling defaults, or explicitly pass 10 steps and guidance 1.0. ## 3. Offline generation The default request produces a muxed video and audio file: ```bash theme={null} sglang generate \ --model-path kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers \ --performance-mode speed --attention-backend fa --warmup-mode off \ --prompt "A woman chops vegetables in a sunlit kitchen, with soft jazz playing." \ --seed 42 --save-output ``` Add `--image-path /path/to/reference.png` for image conditioning. On RTX PRO 6000, use `--attention-backend torch_sdpa` in the CLI or `attention_backend="torch_sdpa"` in Python. For two B200 GPUs, add `--num-gpus 2 --tp-size 1 --ulysses-degree 2 --ring-degree 1`. To use Pro, change the model path; its defaults select 50 steps and guidance 5.0. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator generator = DiffGenerator.from_pretrained( model_path="kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers", num_gpus=1, performance_mode="speed", attention_backend="fa", warmup_mode="off", ) try: result = generator.generate(sampling_params_kwargs={ "prompt": "A woman chops vegetables in a sunlit kitchen, with soft jazz playing.", "seed": 42, "save_output": True, "output_path": "kandinsky6_samples", "return_file_paths_only": False, "return_frames": True, }) print(result.output_file_path) finally: generator.shutdown() ``` `result.frames` contains RGB frames and `result.audio` contains the decoded waveform. Add `image_path` to the sampling parameters for a reference image. Omit the two return-data flags when only the output file is needed. ## 4. Parallelism and memory * **Ulysses**: full-size Pro-distill generation on two B200 or four GB300 GPUs matches single-GPU output on the same hardware, including every decoded RGB frame and the audio waveform. It shards the video stream; audio and text streams remain replicated. Video K/V is gathered for audio-to-video attention. * **CFG parallel**: Pro can distribute its conditional and unconditional branches with `--num-gpus 2 --ulysses-degree 1 --enable-cfg-parallel`. This is a different topology from Ulysses. It does not accelerate Pro-distill, which has no guidance branch. * **TP and Ring**: implemented, but full-model outputs differ from the single-GPU baseline. Do not treat them as verified bitwise-equivalent modes. Ring2, Ring4, and TP2 x Ring2 complete full-size Pro-distill generation on GB300 and repeat exactly within each topology. Ring2 serving also covers image conditioning, Cache-DiT, and dynamic/merged LoRA restoration. Use `--num-gpus 2 --ulysses-degree 1 --ring-degree 2` to select Ring2; validate both video and audio quality before deployment. * **TP x SP**: four-GB300 TP2 x Ulysses2 full generation is tested. It reduces weight memory but was slower than Ulysses4 and produced different video and audio. Prefer Ulysses when memory permits; validate quality before deploying TP. * **FSDP + Ulysses**: add `--use-fsdp-inference true` to shard DiT weights without sharding linear reductions. Four-GB300 Ulysses4 generation matches resident Ulysses4 in every decoded RGB frame and the audio waveform. This is a lower-memory alternative with additional weight-gather latency, not a faster default. Request-level Cache-DiT switching is also tested with FSDP. * **Encoder parallelism**: `--encoder-parallel fold` shards Reason1 and CLIP across the DiT replica, including a Ulysses-only replica. Four-GPU generation is tested, but folding changes reductions and the generated video and audio. Keep `auto` for the baseline recipes and validate quality before forcing folding. See [Encoder parallelism](/docs/sglang-diffusion/encoder_parallel). * **Offload**: keep components resident when memory allows. Use `--cpu-offload-components transformer text_encoder vae` for whole-component offload, or `--layerwise-offload-components all` for layerwise offload. Layerwise offload includes both DiT text towers and the audio VAE's encoder, decoder, and vocoder blocks. Use component-specific residency settings to keep selected components resident. Both require host RAM and add transfers; lower device memory does not imply lower total memory or lower latency. The picker's **Server** tab exposes attention, weight placement and offload components. With DiT layerwise offload, you can also tune prefetch depth and resident layers using `--dit-offload-prefetch-size` and `--dit-layerwise-resident-layers`. Increasing either uses more VRAM; these non-default settings are unverified tuning choices, not faster recommendations. Component offload moves whole components; layerwise offload streams supported blocks. They are alternatives to FSDP in this picker, not additive presets. ### Deployment limits Use monolithic serving. Encoder/denoiser/decoder disaggregation is not supported. Breakable CUDA graphs are also not enabled for Kandinsky6: the server disables `--enable-breakable-cuda-graph`. Its variable-length text path needs additional handling to preserve the unpadded computation; same-shape graph experiments do not establish support for arbitrary requests. Keep eager execution for the recipes on this page. ### Choosing a topology Pro-distill, batch 1, 768 x 512, 121 frames, 10 steps, BF16/FA, no offload or compilation, on GB300 with 277.5 GiB per GPU: | Topology | Hot E2E (s) | Output-rank peak reserved (GiB) | Matches single-GPU RGB/audio | | - | -: | -: | - | | Single GPU | 33.95 | 80.36 | Reference | | Ulysses4 | 11.33 | 80.91 | Exact | | Ulysses4 + FSDP | 12.13 | 42.54 | Exact | | TP2 x Ulysses2 | 13.99 | 50.97 | No | | TP4 | 17.72 | 35.91 | No | Times are medians of three hot requests after one initial request, with startup warmup disabled and default conditioning caching enabled. They include MP4 saving but exclude model loading. Memory is PyTorch runtime peak reserved on the output rank, not loading peak or the maximum across all GPUs. These results do not establish output quality for TP; TP also changes the text encoder's numerical path. For this workload, FSDP reduces measured memory by 47% versus resident Ulysses4 at a 7% latency cost. Add it when weight memory is the constraint; keep ordinary Ulysses when latency is the priority. ### RTX PRO 6000 and SageAttention3 Pro-distill also runs with resident weights on one 96 GB RTX PRO 6000 Blackwell Server Edition. Use `--attention-backend torch_sdpa` on this SM120 device: the current platform resolves `fa` to SDPA, so requesting `fa` does not establish a FlashAttention benchmark. With [SageAttention3](https://github.com/thu-ml/SageAttention/tree/d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5/sageattention3_blackwell) installed, `--component-attention-backends transformer=sage_attn_3` selects it only for the DiT, keeping the encoders on SDPA. The tested upstream revision requires SM120/121 and rejects B200 SM100. With PyTorch 2.14, its build required `NVCC_APPEND_FLAGS=-std=c++20`; no kernel source or hardware guard was modified. Same full-size workload and timing method as above, on one RTX PRO 6000 with driver 595.84, PyTorch 2.14.1+cu130, Transformers 5.17.0, and Diffusers 0.37.0: | DiT backend | Hot E2E (s) | Runtime peak reserved (GiB) | | - | -: | -: | | SDPA | 133.91 | 80.34 | | SageAttention3 | 105.33 | 82.05 | Sage3 reduced latency by 21.3% but changed the outputs substantially: all-frame PSNR 11.97 dB, mean SSIM 0.375, and raw-audio waveform SNR -2.21 dB versus SDPA. These are paired differences, not perceptual quality scores. Each backend was repeatable across four requests, but Sage3 is **not lossless** and is not the default recipe. Validate both video and audio before choosing it. ### VAE decode The video VAE defaults to tiled decoding, including multi-GPU execution. Its decode group is separate from the DiT's TP and CFG collectives. Explicit `--vae-config.parallel-decode-mode spatial_shard` uses whole-clip decoding with frame-causal attention computed without a quadratic whole-video mask. Whole-clip decoding still needs substantial activation memory. A 768 x 512, 121-frame run on two B200 GPUs completed with the transformer, both text encoders, and audio VAE offloaded, but reached 170.4 GiB output-rank peak reserved. This is not a lower-memory replacement for tiled decode. Tiled and whole-clip decoding use different normalization and attention contexts and are not interchangeable numerical baselines; keep tiled decoding for the validated default output. The audio VAE contains both the mel decoder and vocoder. Current checkpoints include a complete vocoder copy inside `audio_vae`, so this pipeline does not also load the redundant standalone `vocoder` component. CUDA matrix RoPE uses a fused kernel that preserves the eager FP32 rounding boundaries. CPU and autograd paths retain the eager implementation. No sampling parameter or sparse-attention approximation is introduced. The checkpoint's 128-wide Q/K normalization uses native PyTorch FP32 fusion during CUDA inference without changing the stored parameter dtype or offload policy. Other head widths retain the original normalization path. ### Cache-DiT Cache-DiT is opt-in per request through `enable_cache_dit` and `cache_dit_params` in the Python sampling parameters or the JSON `/v1/videos` body. Set `enable_cache_dit: false` to restore uncached execution without restarting. See [Cache-DiT configuration](/docs/sglang-diffusion/cache_dit) for the available knobs. The **Request** tab offers an opt-in Cache-DiT switch for both text and image conditioning. It uses the runtime cache defaults, not a quality-calibrated preset. A short request may finish before cache warmup ends and gain no speedup. The adapter caches both evolving video and audio streams together. Its dynamic decision uses the video residual, not an independent audio-quality metric. Skipping blocks is approximate and can change both outputs substantially; validate video and audio on your own prompts before enabling it. It is not part of the lossless defaults, and should not be combined with breakable CUDA graphs. ### LoRA adapters Use adapters trained for the same Kandinsky6 checkpoint architecture. The native loader accepts PEFT A/B weights with Diffusers or SGLang DiT layer names, including the `transformer.` prefix. Both video and audio branches belong to `transformer`; an adapter can change both outputs. Add `--lora-path /path/to/adapter.safetensors` to load an adapter at startup. To switch an existing server to dynamic LoRA without restarting: ```bash theme={null} curl --fail-with-body http://localhost:30000/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": "kandinsky6", "lora_path": "/path/to/adapter.safetensors", "target": "transformer", "strength": 1.0, "merge_mode": "dynamic" }' ``` The adapter path must be readable by the server. `dynamic` applies the low-rank update during inference; `merge` updates the base weights before inference. These modes may differ numerically. To disable either mode and restore the base model: ```bash theme={null} curl --fail-with-body http://localhost:30000/v1/unmerge_lora_weights \ -H "Content-Type: application/json" -d '{"target": "transformer"}' ``` Adapter selection is server-wide, not isolated per request. Coordinate changes with other clients. Merging offloaded weights retains CPU snapshots of the modified layers for exact restoration; budget for this host memory. Validation uses synthetic adapters to check loading, strength changes, repeatability, and exact restoration of video and audio; it does not establish the quality of a community adapter. See the [model inventory](/docs/sglang-diffusion/compatibility_matrix) and the [super-resolution recipe](/cookbook/diffusion/Kandinsky/Kandinsky6-SR) for the separate video SR pipeline. ## 5. Download a completed video Use the `id` returned by the picker's Request command. The same endpoints serve joint generation and SR. Set the base URL to match your server port: ```python Download theme={null} import json import shutil import time import urllib.request base_url = "http://localhost:30000" video_id = "REPLACE_WITH_RETURNED_ID" deadline = time.monotonic() + 1800 while time.monotonic() < deadline: with urllib.request.urlopen(f"{base_url}/v1/videos/{video_id}", timeout=30) as response: job = json.load(response) if job["status"] == "failed": raise RuntimeError(job) if job["status"] == "completed": with urllib.request.urlopen( f"{base_url}/v1/videos/{video_id}/content", timeout=120 ) as response, open("output.mp4", "wb") as output: shutil.copyfileobj(response, output) break time.sleep(2) else: raise TimeoutError("Video generation did not finish within 30 minutes") ``` After downloading, remove the server-side job when no longer needed: ```bash Cleanup theme={null} curl --fail-with-body -X DELETE http://localhost:30000/v1/videos/REPLACE_WITH_RETURNED_ID ``` ## 6. ComfyUI There is no Kandinsky6 integrated DiT executor in the [ComfyUI plugin](/docs/sglang-diffusion/comfyui). The generic server video node has not been validated for this model's joint audio/video output. Use the HTTP or offline recipes above until that integration is verified. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.