> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Kandinsky6 SR > Deploy Kandinsky 6 video super-resolution with x2, x4 or x2.25 upscaling, source audio and explicit decode speed, memory and numerical tradeoffs. export const config = { modelName: "Kandinsky 6.0 SR", supportedHardware: ["gb300", "b200", "h200", "h100", "rtxpro6000", "rtx5090"], hardware: [{ id: "rtxpro6000", label: "RTX PRO 6000", vram: "96GB", vendor: "blackwell" }, { id: "rtx5090", label: "RTX 5090", vram: "32GB", vendor: "blackwell" }], groupHardware: false, matchDims: [], overlayDims: [{ id: "attention", title: "Attention", scope: "serve", default: "platform", description: "Dense attention. Switching kernels can change rounding; neither choice enables NABLA.", learnMore: "#5-memory-and-parallelism", options: [{ id: "platform", label: "Recipe default", recommended: true, description: "SDPA on RTX; FA on datacenter Blackwell and Hopper." }, { id: "fa", label: "FlashAttention", disabled: s => ["rtxpro6000", "rtx5090"].includes(s.hw), disableReason: "This SM120 platform maps FA to SDPA; select SDPA explicitly instead." }, { id: "torch_sdpa", label: "PyTorch SDPA", soft: s => s.hw !== "rtxpro6000", softReason: "SDPA measurements apply to RTX PRO 6000, not every GPU or workload.", disabled: s => s.topology_mode === "manual" && Number(s.ring_degree) > 1, disableReason: "SDPA does not support Ring rotation." }] }, { id: "decode", title: "VAE decode", scope: "serve", default: "tiles", description: "Whole-tile parallelism preserves the tested serial output; spatial sharding is not bitwise lossless.", learnMore: "#4-decode-speed-memory-and-numerics", options: [{ id: "serial", label: "Serial", description: "Reference decode. Each rank processes all tiles; single-GPU and single-tile baseline.", flags: ["--vae-config.use-parallel-tiling false", "--vae-config.use-parallel-decode false"] }, { id: "tiles", label: "Whole-tile parallel", recommended: true, description: "Distributes complete tiles. Tested RGB/audio are exact versus serial on the same topology. Single-tile inputs fall back to serial.", hints: ["Whole-tile decode matched serial RGB/audio in tested same-topology runs."], flags: ["--vae-config.use-parallel-tiling true", "--vae-config.use-parallel-decode false"] }, { id: "spatial", label: "Spatial shard (rounding changes)", description: "Lower activation memory, not bitwise lossless. GB300 measurement: PSNR 57.51 dB, min sampled SSIM 0.99921, max pixel difference 4/255; audio exact.", hints: ["Not bitwise lossless: spatial decode changes rounding; GB300 PSNR 57.51 dB, max pixel difference 4/255."], flags: ["--vae-config.use-parallel-tiling false", "--vae-config.use-parallel-decode true", "--vae-config.parallel-decode-mode spatial_shard"] }] }, { id: "placement", title: "Weight placement", scope: "serve", default: "resident", description: "Offload reduces GPU weight memory, but adds host memory and transfers.", learnMore: "#5-memory-and-parallelism", options: [{ id: "resident", label: "Resident", recommended: true }, { id: "component", label: "Component offload", soft: true, description: "Transfers each selected component whole around its use. This exact SR recipe is unverified.", flags: s => [`--cpu-offload-components ${s.offload_components}`] }, { id: "offload", label: "Layerwise offload", flags: s => [`--layerwise-offload-components ${s.offload_components}`], description: "Streams supported DiT, KVAE and latent-upscaler blocks. It can be slower than resident execution." }, { id: "fsdp", label: "FSDP", soft: true, softReason: "SR weight sharding has not completed full-checkpoint E2E validation.", description: "Shards DiT weights across GPUs. Selectable for validation, not a verified memory or latency recommendation.", flags: ["--use-fsdp-inference true"], disabled: s => Number(s.gpus_per_node) < 2, disableReason: "Weight sharding requires multiple GPUs." }] }, { id: "offload_components", title: "Offload components", scope: "serve", default: "all", showWhen: s => ["component", "offload"].includes(s.placement), description: "Choose which weights leave GPU memory. Partial selections are unverified recipes.", learnMore: "#5-memory-and-parallelism", options: [{ id: "all", label: "All components" }, { id: "transformer", label: "DiT", soft: true }, { id: "vae", label: "VAE", soft: true }, { id: "latent_upscaler", label: "Latent upscaler", soft: true }, { id: "transformer vae", label: "DiT + VAE", soft: true }, { id: "transformer latent_upscaler", label: "DiT + upscaler", soft: true }, { id: "vae latent_upscaler", label: "VAE + upscaler", soft: true }] }, { id: "prefetch", title: "DiT prefetch", scope: "serve", kind: "number", min: 1, max: 4, default: 1, unit: "layers", options: [], showWhen: s => s.placement === "offload" && (s.offload_components === "all" || s.offload_components.includes("transformer")), verifiedWhen: s => Number(s.prefetch) === 1, description: "More prefetched layers use more VRAM and may overlap transfers better. Values above 1 are unverified here.", learnMore: "#5-memory-and-parallelism" }, { id: "resident_layers", title: "DiT resident layers", scope: "serve", kind: "number", min: 0, max: 8, default: 0, unit: "layers", options: [], showWhen: s => s.placement === "offload" && (s.offload_components === "all" || s.offload_components.includes("transformer")), verifiedWhen: s => Number(s.resident_layers) === 0, description: "Keep leading DiT layers on GPU during the request. Trades VRAM for fewer transfers; nonzero recipes are unverified.", learnMore: "#5-memory-and-parallelism" }, { id: "scale", title: "Upscale factor", scope: "request", default: "2", description: "Choose output scale, not quality. All use the two-step distilled checkpoint.", learnMore: "#request-parameters", options: [{ id: "2", label: "2x" }, { id: "4", label: "4x" }, { id: "2.25", label: "2.25x" }] }, { id: "tile_batch", title: "Tile batch", scope: "request", default: "1", description: "Changing tile batching changes seeded noise and therefore the output.", options: [{ id: "1", label: "1", recommended: true }, { id: "2", label: "2 (different output)", description: "Uses more activation memory. Not an output-preserving acceleration." }] }], commandBuilder: { defaultSelection: { hw: "gb300", nodes: 1, gpus_per_node: 4, topology_mode: "auto", tp_size: 1, ulysses_degree: 4, ring_degree: 1 }, resource: { limits: { nodes: { min: 1, max: 1 }, gpus_per_node: { min: 1, max: 8 } }, verifiedRecipes: [{ id: "gb300-ulysses4-offline", hw: "gb300", nodes: 1, gpus_per_node: 4, tp_size: 1, ulysses_degree: 4, ring_degree: 1, placement: "resident", decode: "tiles", default: true, unverified: true }, ...["b200", "h200", "h100", "rtxpro6000", "rtx5090"].map(hw => ({ id: `${hw}-single-gpu`, hw, nodes: 1, gpus_per_node: 1, tp_size: 1, ulysses_degree: 1, ring_degree: 1, placement: hw === "rtx5090" ? "offload" : "resident", decode: "serial", default: true, unverified: hw !== "rtxpro6000" })), { id: "rtxpro6000-ulysses2-tiles", hw: "rtxpro6000", nodes: 1, gpus_per_node: 2, tp_size: 1, ulysses_degree: 2, ring_degree: 1, placement: "resident", decode: "tiles" }], autoTopology: s => ({ tp_size: 1, ulysses_degree: Number(s.gpus_per_node) === 8 ? 4 : Number(s.gpus_per_node), ring_degree: Number(s.gpus_per_node) === 8 ? 2 : 1 }), validateTopology: (s, t) => { const errors = []; const degrees = [t.tp_size, t.ulysses_degree, t.ring_degree]; if (!degrees.every(n => Number.isInteger(n) && n >= 1)) errors.push("Parallel degrees must be positive integers."); if (Number(s.nodes) !== 1) errors.push("This picker covers one node."); if (degrees.reduce((a, b) => a * b, 1) !== Number(s.gpus_per_node)) errors.push("GPU count must equal TP times Ulysses times Ring."); if (28 % (t.tp_size * t.ulysses_degree)) errors.push("TP times Ulysses must divide the 28 DiT heads."); if (s.placement === "fsdp" && Number(s.gpus_per_node) < 2) errors.push("FSDP requires multiple GPUs."); if ((s.attention === "torch_sdpa" || ["rtxpro6000", "rtx5090"].includes(s.hw)) && t.ring_degree > 1) errors.push("SDPA does not support Ring rotation."); if (s.attention === "fa" && ["rtxpro6000", "rtx5090"].includes(s.hw)) errors.push("FA resolves to SDPA on this SM120 platform; select SDPA instead."); return errors; } }, resolveDeployment: s => { const resource = config.commandBuilder.resource; const topology = s.topology_mode === "manual" ? { tp_size: Number(s.tp_size), ulysses_degree: Number(s.ulysses_degree), ring_degree: Number(s.ring_degree) } : resource.autoTopology(s); const errors = resource.validateTopology(s, topology); const platformAttention = ["rtxpro6000", "rtx5090"].includes(s.hw) ? "torch_sdpa" : "fa"; const attention = s.attention === "platform" ? platformAttention : s.attention; const streamingDit = s.placement === "offload" && (s.offload_components === "all" || s.offload_components.includes("transformer")); const gb300Served = s.hw === "gb300" && attention === "fa" && (topology.tp_size === 2 && topology.ulysses_degree === 2 && topology.ring_degree === 1 && s.placement === "resident" && s.decode !== "spatial" || topology.tp_size === 1 && topology.ulysses_degree === 1 && topology.ring_degree === 2 && s.placement === "offload" && s.offload_components === "all" && Number(s.prefetch) === 1 && Number(s.resident_layers) === 0 && s.decode !== "spatial"); const rtxServed = s.hw === "rtxpro6000" && attention === "torch_sdpa" && s.placement === "resident" && topology.tp_size === 1 && topology.ring_degree === 1 && (topology.ulysses_degree === 1 && s.decode === "serial" || topology.ulysses_degree === 2 && s.decode === "tiles"); const served = errors.length === 0 && (gb300Served || rtxServed); const warnings = ["Verification describes execution, not lossless output. Arbitrary uploaded videos are outside the fixed HTTP test workload."]; if (!served) warnings.push("This exact HTTP recipe is unverified. Offline timing does not verify every server and uploaded-video combination."); if (!["gb300", "rtxpro6000"].includes(s.hw)) warnings.push("Extrapolated hardware recipe: no SR latency or peak-memory measurement on this GPU. Validate a short clip before scaling up."); if (attention !== platformAttention) warnings.push("This backend override is a different numerical path; its full-checkpoint output and performance are unverified."); if (s.hw === "rtx5090") warnings.push("32GB VRAM: start with layerwise offload, tile batch 1 and a short 384 x 256 source. Offload does not bound activation memory; full-size video may still OOM."); if (Number(s.gpus_per_node) > 4) warnings.push("More than four GPUs is extrapolated. Eight GPUs use Ulysses4 x Ring2 with FA; Ulysses8 cannot divide 28 heads."); if (s.hw === "rtxpro6000" && Number(s.gpus_per_node) > 2) warnings.push("RTX PRO 6000 measurements cover one or two GPUs only; larger GPU counts are extrapolated."); if (["component", "offload"].includes(s.placement)) warnings.push("Offload uses host RAM and transfer bandwidth; prefetch and resident layers additionally consume VRAM."); if (s.decode === "spatial") warnings.push("Not bitwise lossless: spatial decode changes rounding. Tested PSNR 57.51 dB, max pixel difference 4/255; validate your video."); if (s.decode === "tiles") warnings.push("Whole-tile decode matched serial RGB/audio exactly on tested workloads within each topology, not across different TP/Ring settings."); if (topology.tp_size > 1 || topology.ring_degree > 1) warnings.push("TP/Ring can change output numerics independently of the VAE decode mode."); if (s.tile_batch !== "1") warnings.push("Tile batch 2 changes seeded noise and output; it is not lossless."); if (Number(s.gpus_per_node) === 1 && s.decode !== "serial") warnings.push("One GPU uses serial decoding; this selection provides no parallel speedup."); return { match: { hw: s.hw }, nnodes: Number(s.nodes), verified: false, flags: ["--model-path {{MODEL_NAME}}", `--num-gpus ${Number(s.gpus_per_node)}`, `--tp-size ${topology.tp_size}`, `--ulysses-degree ${topology.ulysses_degree}`, `--ring-degree ${topology.ring_degree}`, "--performance-mode speed", `--attention-backend ${attention}`, "--warmup-mode off", "--host {{HOST_IP}}", "--port {{PORT}}", ...streamingDit && Number(s.prefetch) !== 1 ? [`--dit-offload-prefetch-size ${s.prefetch}`] : [], ...streamingDit && Number(s.resident_layers) !== 0 ? [`--dit-layerwise-resident-layers ${s.resident_layers}`] : []], builder: { topology, resolvedSettings: { attention: attention === "fa" ? "FlashAttention" : "PyTorch SDPA", prefetch: `${s.prefetch} ${Number(s.prefetch) === 1 ? "layer" : "layers"}`, resident_layers: `${s.resident_layers} ${Number(s.resident_layers) === 1 ? "layer" : "layers"}` }, topologySummary: `TP ${topology.tp_size}, Ulysses ${topology.ulysses_degree}, Ring ${topology.ring_degree}`, errors, warnings, verification: { serve: errors.length ? "error" : served ? "verified" : "unverified", request: errors.length ? "error" : "unverified" } } }; } }, modelNames: { default: "kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers" }, placeholders: { HOST_IP: { target: "command", label: "Bind host", default: "0.0.0.0" }, PORT: { target: "command", label: "Bind port", default: "30000" }, CURL_HOST: { target: "curl", label: "Server host", default: "localhost" }, CURL_PORT: { target: "curl", label: "Server port", default: "30000" }, INPUT_VIDEO: { target: "curl", label: "Source video", default: "/path/to/input.mp4" } }, curl: s => `curl -sS --fail-with-body http://{{CURL_HOST}}:{{CURL_PORT}}/v1/videos \\ -F "video_reference=@{{INPUT_VIDEO}};type=video/mp4" \\ -F sr_resolution_scale=${s.scale} \\ -F sr_tiles_batch_size=${s.tile_batch} \\ -F num_inference_steps=2 \\ -F seed=42`, runModes: () => ["python"], showPlaygroundLink: false, cells: [] }; export const Deployment = ({config, benchmarks}) => { if (!config) { return
Deployment: missing config prop
; } const AMD_RDMA_DOCKER_FLAGS = ["--device /dev/infiniband", "--cap-add IPC_LOCK", "--ulimit memlock=-1", "--ulimit stack=67108864", "--ulimit nofile=1048576:1048576"]; const HARDWARE_CATALOG = { blackwell: [{ id: "b300", label: "B300", vram: "288GB" }, { id: "gb300", label: "GB300", vram: "288GB" }, { id: "b200", label: "B200", vram: "192GB" }, { id: "gb200", label: "GB200", vram: "192GB" }, { id: "dgx-spark", label: "DGX Spark", vram: "128GB", multiNodeDockerFlags: ["--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband"] }], hopper: [{ id: "h200", label: "H200", vram: "141GB" }, { id: "h100", label: "H100", vram: "80GB" }, { id: "h20-3e", label: "H20-3e", vram: "141GB" }, { id: "h800", label: "H800", vram: "80GB" }], amd: [{ id: "mi300x", label: "MI300X", vram: "192GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi325x", label: "MI325X", vram: "256GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi350x", label: "MI350X", vram: "288GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, { id: "mi355x", label: "MI355X", vram: "288GB", multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }], npu: [{ id: "a3", label: "A3 Series", vram: "64GB/die", npuDevices: 16 }, { id: "a5", label: "950PR/DT Series", vram: "128GB", npuDevices: 8 }] }; const makeStyles = isDark => ({ container: { maxWidth: "900px", margin: "0 auto", display: "flex", flexDirection: "column", gap: "3px" }, card: { padding: "5px 10px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", display: "flex", alignItems: "center", gap: "10px", background: isDark ? "#1f2937" : "#fff" }, cardColumn: { padding: "5px 10px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", display: "flex", flexDirection: "column", gap: "4px", background: isDark ? "#1f2937" : "#fff" }, title: { fontSize: "12px", fontWeight: "600", minWidth: "108px", flexShrink: 0, color: isDark ? "#e5e7eb" : "inherit" }, vendorRow: { display: "flex", alignItems: "center", gap: "6px" }, vendorLabel: { fontSize: "10px", fontWeight: "600", color: isDark ? "#9ca3af" : "#6b7280", width: "68px", flexShrink: 0, textTransform: "uppercase", letterSpacing: "0.04em" }, itemsGrid: () => ({ display: "grid", gridTemplateColumns: "repeat(auto-fit, minmax(72px, 1fr))", gap: "4px", flex: 1 }), labelBase: { padding: "2px 8px", border: `1px solid ${isDark ? "#9ca3af" : "#d1d5db"}`, borderRadius: "3px", cursor: "pointer", display: "inline-flex", flexDirection: "column", alignItems: "center", justifyContent: "center", fontWeight: "500", fontSize: "12px", transition: "all 0.2s", userSelect: "none", minHeight: "26px", textAlign: "center", background: isDark ? "#374151" : "#fff", color: isDark ? "#e5e7eb" : "inherit" }, checked: { background: "#D45D44", color: "white", borderColor: "#D45D44" }, disabled: { cursor: "not-allowed", opacity: 0.4 }, subtitle: { display: "block", fontSize: "9px", marginTop: "1px", lineHeight: "1.1", opacity: 0.7 }, commandWrap: { position: "relative", flex: 1, background: isDark ? "#111827" : "#f5f5f5", borderRadius: "6px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, overflow: "hidden" }, commandHeader: { display: "flex", flexWrap: "wrap", justifyContent: "space-between", alignItems: "center", gap: "6px 10px", padding: "6px 10px", borderBottom: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, background: isDark ? "#1f2937" : "#fafafa" }, commandPre: { padding: "12px 16px", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontSize: "12px", lineHeight: "1.5", color: isDark ? "#e5e7eb" : "#374151", whiteSpace: "pre-wrap", overflowX: "auto", margin: 0 }, mtpWarn: { margin: "8px 0 0", padding: "8px 12px", borderRadius: "8px", fontSize: "12px", lineHeight: "1.45", background: isDark ? "#78350f" : "#fef3c7", color: isDark ? "#fde68a" : "#92400e", border: `1px solid ${isDark ? "#92400e" : "#fcd34d"}` }, badge: status => ({ display: "inline-flex", alignItems: "center", gap: "6px", padding: "2px 8px", borderRadius: "10px", background: ({ verified: isDark ? "#064e3b" : "#d1fae5", "in-progress": isDark ? "#1e3a8a" : "#dbeafe", unverified: isDark ? "#78350f" : "#fef3c7" })[verifyStatusOf(status)], color: ({ verified: isDark ? "#a7f3d0" : "#065f46", "in-progress": isDark ? "#bfdbfe" : "#1e40af", unverified: isDark ? "#fde68a" : "#92400e" })[verifyStatusOf(status)], fontSize: "11px", fontWeight: 600, whiteSpace: "nowrap" }), badgeDot: status => ({ width: "8px", height: "8px", borderRadius: "50%", background: ({ verified: "#10b981", "in-progress": "#3b82f6", unverified: "#f59e0b" })[verifyStatusOf(status)] }), iconButton: { padding: "4px 10px", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#374151", fontSize: "11px", fontWeight: 500, cursor: "pointer", display: "inline-flex", alignItems: "center", gap: "4px" }, iconRow: { display: "inline-flex", flexWrap: "wrap", gap: "6px" }, runModeWrap: { display: "inline-flex", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "10px", overflow: "hidden", fontSize: "11px", fontWeight: 600, userSelect: "none" }, runModeChip: active => ({ padding: "2px 10px", cursor: "pointer", background: active ? isDark ? "#1f2937" : "#fff" : "transparent", color: active ? isDark ? "#e5e7eb" : "#111827" : isDark ? "#9ca3af" : "#6b7280", borderRight: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}` }), runModeChipLast: active => ({ padding: "2px 10px", cursor: "pointer", background: active ? isDark ? "#1f2937" : "#fff" : "transparent", color: active ? isDark ? "#e5e7eb" : "#111827" : isDark ? "#9ca3af" : "#6b7280" }), headerLeft: { display: "inline-flex", flexWrap: "wrap", alignItems: "center", gap: "8px" }, modalBackdrop: { position: "fixed", inset: 0, background: "rgba(0,0,0,0.5)", display: "flex", alignItems: "center", justifyContent: "center", zIndex: 9999 }, modalBox: { background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#111827", borderRadius: "8px", padding: "20px", maxWidth: "720px", width: "92%", maxHeight: "85vh", overflowY: "auto", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, boxShadow: "0 10px 25px rgba(0,0,0,0.25)" }, modalHeader: { display: "flex", justifyContent: "space-between", alignItems: "center", marginBottom: "12px" }, modalTitle: { fontSize: "15px", fontWeight: 600 }, modalCloseBtn: { background: "transparent", border: "none", color: "inherit", fontSize: "20px", cursor: "pointer", padding: "0 6px", lineHeight: 1 }, formField: { display: "flex", flexDirection: "column", gap: "4px", marginBottom: "10px" }, formLabel: { fontSize: "12px", fontWeight: 500, color: isDark ? "#9ca3af" : "#4b5563" }, formInput: { padding: "6px 10px", fontSize: "13px", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#111827" : "#fff", color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, sectionHeading: { fontSize: "12px", fontWeight: 600, textTransform: "uppercase", letterSpacing: "0.04em", color: isDark ? "#9ca3af" : "#6b7280", margin: "12px 0 6px 0" }, primaryBtn: { padding: "6px 14px", background: "#D45D44", color: "white", border: "none", borderRadius: "4px", cursor: "pointer", fontSize: "13px", fontWeight: 500 }, benchCard: { padding: "8px 12px", border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderLeft: `3px solid ${isDark ? "#E85D4D" : "#D45D44"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", display: "flex", flexDirection: "column", gap: "8px" }, benchHeader: { display: "flex", flexWrap: "wrap", alignItems: "baseline", justifyContent: "space-between", gap: "6px 12px" }, benchTitle: { fontSize: "13px", fontWeight: 600, color: isDark ? "#e5e7eb" : "inherit" }, benchVersion: { fontSize: "11px", color: isDark ? "#9ca3af" : "#6b7280" }, benchHeaderRight: { display: "flex", flexWrap: "wrap", alignItems: "center", gap: "6px 10px", flexShrink: 0 }, benchChipRow: { display: "flex", alignItems: "center", gap: "6px", flexWrap: "wrap", margin: "2px 0 8px" }, benchChip: { padding: "2px 10px", fontSize: "12px", cursor: "pointer", border: `1px solid ${isDark ? "#4b5563" : "#d1d5db"}`, borderRadius: "4px", background: isDark ? "#1f2937" : "#fff", color: isDark ? "#e5e7eb" : "#374151", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, benchChipActive: { background: "#D45D44", color: "white", borderColor: "#D45D44" }, benchBlock: { border: `1px solid ${isDark ? "#374151" : "#e5e7eb"}`, borderRadius: "4px", padding: "8px 10px", background: isDark ? "#111827" : "#fafafa" }, benchBlockTitle: { fontSize: "11px", fontWeight: 600, textTransform: "uppercase", letterSpacing: "0.04em", color: isDark ? "#9ca3af" : "#6b7280", marginBottom: "4px" }, benchWorkload: { fontSize: "11px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280", marginBottom: "6px", lineHeight: "1.3" }, benchRow: { display: "flex", justifyContent: "space-between", fontSize: "12px", padding: "2px 0" }, benchKey: { color: isDark ? "#9ca3af" : "#6b7280" }, benchVal: { color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontWeight: 500 }, benchNotes: { fontSize: "11px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280" }, benchLegend: { fontSize: "10px", fontStyle: "italic", color: isDark ? "#6b7280" : "#9ca3af", marginTop: "6px", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace" }, benchEmpty: { fontSize: "12px", fontStyle: "italic", color: isDark ? "#9ca3af" : "#6b7280" }, benchTableScroll: { overflowX: "auto", overflowY: "hidden", marginTop: "4px" }, benchTable: { display: "grid", columnGap: 0, rowGap: "3px", alignItems: "baseline", minWidth: "max-content" }, benchTableHead: { textAlign: "right", fontWeight: 500, fontSize: "11px", color: isDark ? "#9ca3af" : "#6b7280", paddingLeft: "16px", paddingBottom: "4px", whiteSpace: "nowrap" }, benchTableCornerHead: { paddingBottom: "4px", position: "sticky", left: 0, background: isDark ? "#111827" : "#fafafa", zIndex: 1, alignSelf: "stretch" }, benchTableSeparator: { gridColumn: "1 / -1", height: "1px", background: isDark ? "#374151" : "#e5e7eb", marginTop: "-3px" }, benchTableLabel: { textAlign: "left", fontSize: "12px", color: isDark ? "#9ca3af" : "#6b7280", whiteSpace: "nowrap", position: "sticky", left: 0, background: isDark ? "#111827" : "#fafafa", zIndex: 1 }, benchTableValue: { textAlign: "right", fontSize: "12px", color: isDark ? "#e5e7eb" : "#111827", fontFamily: "'Menlo', 'Monaco', 'Courier New', monospace", fontWeight: 500, paddingLeft: "16px", whiteSpace: "nowrap" }, benchTableValueMissing: { color: isDark ? "#6b7280" : "#9ca3af" } }); const VERIFY_LABEL = { verified: "Verified", "in-progress": "Final Verification In Progress", unverified: "Not Verified" }; const verifyStatusOf = v => typeof v === "string" ? VERIFY_LABEL[v] ? v : "unverified" : v ? "verified" : "unverified"; const cellVerifyStatus = (c, sel) => { if (!c) return "unverified"; const v = typeof c.verificationStatus === "function" ? c.verificationStatus(sel) : c.verificationStatus; return verifyStatusOf(v ?? c.verified); }; const LEGACY_MATCH_DIMS = [{ id: "variant", title: "Model Variant", optionsKey: "variants" }, { id: "quant", title: "Quantization", optionsKey: "quantizations" }, { id: "strategy", title: "Strategy", optionsKey: "strategies" }, { id: "nodes", title: "Nodes", optionsKey: "nodesOptions" }]; const matchDimSpecs = (config.matchDims || LEGACY_MATCH_DIMS).map(d => ({ ...d, options: d.options || config[d.optionsKey] || [] })); const overlayDimSpecs = config.overlayDims || []; const commandBuilder = config.commandBuilder || null; const DIMENSIONS = ["hw", ...matchDimSpecs.map(d => d.id)]; const optionVisible = (opt, sel) => typeof opt.showWhen !== "function" || opt.showWhen(sel); const optionDisabled = (opt, sel) => typeof opt.disabled === "function" ? opt.disabled(sel) : !!opt.disabled; const visibleOptions = (spec, sel) => (spec.options || []).filter(o => optionVisible(o, sel)); const rowVisible = (spec, sel) => (typeof spec.showWhen !== "function" || spec.showWhen(sel)) && visibleOptions(spec, sel).length > 0; const overlayPick = sel => { const picked = []; for (const spec of config.overlayDims || []) { if (!rowVisible(spec, sel)) continue; const opt = (spec.options || []).find(o => o.id === sel[spec.id]); if (opt && !optionDisabled(opt, sel)) picked.push(opt); } return picked; }; const overlayPart = (sel, key) => { const out = []; for (const opt of overlayPick(sel)) { const add = typeof opt[key] === "function" ? opt[key](sel) : opt[key]; if (add) out.push(...add); } return out; }; const overlayCompose = (cellFlags, sel) => { const strip = overlayPart(sel, "stripPrefixes"); const add = overlayPart(sel, "flags"); if (!strip.length) return [...cellFlags || [], ...add]; const used = new Set(); const replacementsFor = tok => { const out = []; add.forEach((f, i) => { if (used.has(i) || f.split(/[\s=]/)[0] !== tok) return; used.add(i); out.push(f); }); return out; }; const out = []; for (const f of cellFlags || []) { const tok = f.split(/[\s=]/)[0]; if (!strip.includes(tok)) out.push(f); else out.push(...replacementsFor(tok)); } add.forEach((f, i) => { if (!used.has(i)) out.push(f); }); return out; }; const optionSoft = (opt, sel) => typeof opt.soft === "function" ? opt.soft(sel) : !!opt.soft; const findCell = (cells, sel) => cells.find(c => DIMENSIONS.every(d => c.match[d] === sel[d])); const findBenchmark = (list, sel) => { const hits = (list || []).filter(b => Object.entries(b.match || ({})).every(([k, v]) => sel[k] === v)); return hits.sort((a, b) => Object.keys(b.match).length - Object.keys(a.match).length)[0] || null; }; const normalizeSpeed = speed => { if (!speed) return []; return Array.isArray(speed) ? speed : [speed]; }; const effectiveAccuracy = (entry, sel) => entry ? { ...config.defaultAccuracy && config.defaultAccuracy[sel.variant] || ({}), ...entry.accuracy || ({}) } : {}; const benchmarkIsEmpty = (entry, accuracy) => { for (const m of normalizeSpeed(entry && entry.speed)) { if (m && typeof m === "object") { for (const [key, v] of Object.entries(m)) { if (key === "workload") continue; if (v !== null && v !== undefined) return false; } } } if (accuracy && typeof accuracy === "object") { for (const v of Object.values(accuracy)) { if (v !== null && v !== undefined) return false; } } return true; }; const isOptionAvailable = (cells, sel, dim, value) => { const idx = DIMENSIONS.indexOf(dim); const higher = DIMENSIONS.slice(0, idx); return cells.some(c => c.match[dim] === value && higher.every(d => c.match[d] === sel[d])); }; const snapToValidCell = (cells, sel, dim, value) => { const idx = DIMENSIONS.indexOf(dim); const higher = DIMENSIONS.slice(0, idx); const lower = DIMENSIONS.slice(idx + 1); let best = null, bestLowerMatches = -1; for (const c of cells) { if (c.match[dim] !== value) continue; if (!higher.every(d => c.match[d] === sel[d])) continue; let s = 0; for (const d of lower) if (c.match[d] === sel[d]) s++; if (s > bestLowerMatches) { bestLowerMatches = s; best = c; } } if (!best) return sel; const next = { ...sel, [dim]: value }; for (const d of lower) next[d] = best.match[d]; return next; }; const validateSelection = (cells, parsed) => { const valid = {}; for (const dim of DIMENSIONS) { const want = parsed[dim]; const works = cells.some(c => c.match[dim] === want && DIMENSIONS.slice(0, DIMENSIONS.indexOf(dim)).every(d => c.match[d] === valid[d])); if (works) { valid[dim] = want; } else { const fallback = cells.find(c => DIMENSIONS.slice(0, DIMENSIONS.indexOf(dim)).every(d => c.match[d] === valid[d])); valid[dim] = fallback ? fallback.match[dim] : want; } } for (const spec of overlayDimSpecs) { const want = parsed[spec.id]; const opts = spec.options || []; const picked = opts.some(o => o.id === want) ? want : (spec.default ?? (opts[0] && opts[0].id)) ?? ""; const withPick = { ...valid, [spec.id]: picked }; const usable = visibleOptions(spec, withPick).filter(o => !optionDisabled(o, withPick)); valid[spec.id] = usable.some(o => o.id === picked) ? picked : (usable[0] && usable[0].id) ?? picked; } return valid; }; const resolveModelName = sel => { const keys = [`${sel.hw}|${sel.variant}|${sel.quant}`, `${sel.variant}|${sel.quant}`, `${sel.hw}|${sel.quant}`, sel.quant, sel.hw, "default"]; for (const k of keys) { const hit = config.modelNames[k]; if (hit) return hit; } return ""; }; const interpolate = (text, env, modelName) => text.replace(/{{(\w+)}}/g, (_, key) => key === "MODEL_NAME" ? modelName : env[key] ?? `{{${key}}}`); const parseNnodes = id => { if (Number.isInteger(id)) return id; if ((/^\d+$/).test(id || "")) return parseInt(id, 10); if (id === "single") return 1; const m = (/^multi-(\d+)$/).exec(id || ""); return m ? parseInt(m[1], 10) : 1; }; const cellNnodes = (cell, sel) => sel.nodes !== undefined ? parseNnodes(sel.nodes) : cell.nnodes || 1; const PD_SERVE_PORTS = { prefill: 30000, decode: 30100 }; const overlayEnv = sel => overlayPart(sel, "env"); const overlayHints = sel => overlayPart(sel, "hints"); const renderCommand = (cell, sel, envValues, mode = "python") => { if (!cell) return "# No command available for the current selection."; const modelName = resolveModelName(sel); const nnodes = cellNnodes(cell, sel); const multinode = nnodes > 1; const cellEnv = [...cell.env || [], ...overlayEnv(sel)]; const flags = overlayCompose(cell.flags, sel); if (multinode) { const PARALLELISM_ANCHORS = new Set(["--attn-dp-size", "--attention-data-parallel-size", "--dp-size", "--dp", "--tp-size", "--tp", "--sp-degree", "--ulysses-degree", "--ring-degree"]); let i = flags.reduce((last, flag, index) => PARALLELISM_ANCHORS.has(flag.split(/[\s=]/)[0]) ? index : last, -1); if (i === -1) i = flags.findIndex(f => f.startsWith("--model-path")); flags.splice(i + 1, 0, `--nnodes ${nnodes}`, `--node-rank {{NODE_RANK}}`, `--dist-init-addr {{NODE0_IP}}:20000`); } const pdServePort = PD_SERVE_PORTS[sel.pdMode]; if (pdServePort !== undefined) { for (let j = 0; j < flags.length; j++) { if (flags[j].split(/[\s=]/)[0] === "--port") { flags[j] = `--port ${pdServePort}`; } } } let cmd; if (mode === "docker") { const di = config.dockerImages || ({}); const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`] || di[`${sel.variant}|${sel.quant}`] || di[`${sel.hw}|${sel.quant}|${sel.strategy}`] || di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev"; const dockerRunCommand = typeof config.dockerRunCommand === "function" ? config.dockerRunCommand(sel) : config.dockerRunCommand || "sglang serve"; const portFlag = flags.find(x => x.split(/[\s=]/)[0] === "--port"); const servePort = portFlag ? portFlag.slice(("--port").length).trim() : "{{PORT}}"; const hostNetwork = multinode || typeof config.dockerHostNetworkWhen === "function" && config.dockerHostNetworkWhen(sel, { flags, env: cellEnv }); const vendorOf = hwId => { for (const [vendor, list] of Object.entries(HARDWARE_CATALOG)) { if (list.some(h => h.id === hwId)) return vendor; } const extra = (config.hardware || []).find(h => h.id === hwId); return extra && extra.vendor || "nvidia"; }; const catalogEntryOf = hwId => { const extra = (config.hardware || []).find(h => h.id === hwId); if (extra) return extra; for (const list of Object.values(HARDWARE_CATALOG)) { const hit = list.find(h => h.id === hwId); if (hit) return hit; } return null; }; const fabricFlagsOf = hwId => (catalogEntryOf(hwId) || ({})).multiNodeDockerFlags || []; const davinciLines = devices => { const lines = []; for (let i = 0; i < devices; i += 4) { const group = []; for (let k = i; k < Math.min(i + 4, devices); k++) { group.push(`--device=/dev/davinci${k}`); } lines.push(" " + group.join(" ")); } return lines; }; const gpuAccessLines = vendorOf(sel.hw) === "amd" ? ["docker run", " --device=/dev/kfd --device=/dev/dri", " --group-add video", " --cap-add=SYS_PTRACE --security-opt seccomp=unconfined", " --shm-size 32g"] : vendorOf(sel.hw) === "npu" ? ["docker run --privileged --shm-size=16g", ...davinciLines((catalogEntryOf(sel.hw) || ({})).npuDevices || 16), " --device=/dev/davinci_manager", " --device=/dev/hisi_hdc", " -v /usr/local/sbin:/usr/local/sbin", " -v /usr/local/Ascend/driver:/usr/local/Ascend/driver", " -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware", " -v /etc/ascend_install.info:/etc/ascend_install.info", " -v /var/queue_schedule:/var/queue_schedule", " -v ~/.cache/:/root/.cache/"] : ["docker run --gpus all", " --shm-size 32g"]; const dockerLines = [...gpuAccessLines, hostNetwork ? " --network host" : ` -p ${servePort}:${servePort}`, ...multinode ? fabricFlagsOf(sel.hw).map(f => " " + f) : [], ...vendorOf(sel.hw) === "npu" ? [] : [" -v ~/.cache/huggingface:/root/.cache/huggingface"], ...(config.dockerMounts || []).map(mount => ` -v ${mount}`), ...config.placeholders && config.placeholders.HF_TOKEN ? [` --env "HF_TOKEN={{HF_TOKEN}}"`] : [], ...cellEnv.map(e => ` --env ${e}`), " --ipc=host", ` ${image}`, ` ${dockerRunCommand}`, ...flags.map(f => " " + f)]; cmd = dockerLines.join(" \\\n"); } else { const flagBlock = flags.map(f => " " + f).join(" \\\n"); const envBlock = cellEnv.length ? cellEnv.join(" \\\n") + " \\\n" : ""; cmd = `${envBlock}sglang serve \\\n${flagBlock}`; } const hintLines = [...overlayHints(sel), ...multinode && config.multiNodeHints && config.multiNodeHints[sel.hw] ? config.multiNodeHints[sel.hw] : []]; if (hintLines.length) { const hint = hintLines.map(line => line.length ? "# " + line : "#").join("\n"); cmd = `${hint}\n${cmd}`; } cmd = interpolate(cmd, envValues, modelName); if (multinode) { const header = `# Multi-node (${nnodes} nodes). Run the same command on every node with:\n` + `# = 0 on the head node, 1..${nnodes - 1} on the others\n` + `# = IP of the head node (reachable from all others)`; cmd = `${header}\n${cmd}`; } return cmd; }; const ACCURACY_LABELS = config.accuracyLabels || []; const renderBenchmarkCard = entry => { const pct = entry && entry.latencyPercentile || config.latencyPercentile || "P50"; const SPEED_LABELS = [["ttft_ms", `TTFT (${pct})`, "ms"], ["tpot_ms", `TPOT (${pct})`, "ms"], ["tokens_per_sec_per_gpu", "throughput per gpu", "tok/s"], ["interactivity", "interactivity", "tokens/s/user", m => m.tpot_ms != null && m.tpot_ms !== 0 ? Math.round(1000 / m.tpot_ms * 10) / 10 : null]]; const WORKLOAD_KEYS = ["dataset", "isl", "osl", "max_concurrency"]; const fmt = (val, unit) => { if (val === null || val === undefined) return null; return `${val}${unit ? " " + unit : ""}`; }; const formatWorkloadParts = (workload, keys) => { if (!workload) return ""; const parts = []; if (keys.has("dataset") && workload.dataset) parts.push(workload.dataset); if (keys.has("isl") || keys.has("osl")) { if (workload.isl != null || workload.osl != null) { parts.push(`in/out=${workload.isl != null ? workload.isl : "?"}/${workload.osl != null ? workload.osl : "?"}`); } } if (keys.has("max_concurrency") && workload.max_concurrency != null) { parts.push(`max-concurrency=${workload.max_concurrency}`); } return parts.join(", "); }; const ALWAYS_PER_COLUMN = new Set(["max_concurrency"]); const ATOMIC_WORKLOAD_GROUPS = [["isl", "osl"]]; const partitionWorkload = measurements => { const shared = new Set(); const differing = new Set(); for (const k of WORKLOAD_KEYS) { const seen = new Set(); let anyPresent = false; for (const m of measurements) { const v = m && m.workload ? m.workload[k] : undefined; if (v != null) anyPresent = true; seen.add(v); } if (!anyPresent) continue; if (ALWAYS_PER_COLUMN.has(k) || seen.size > 1) differing.add(k); else shared.add(k); } for (const group of ATOMIC_WORKLOAD_GROUPS) { if (!group.some(k => differing.has(k))) continue; for (const k of group) if (shared.delete(k)) differing.add(k); } return { shared, differing }; }; const renderBenchTable = ({title, sharedText, colHeaders, rows, colCount, legend}) => { if (rows.length === 0) return null; const showColHeaders = colHeaders.length > 0 && colHeaders.some(h => h !== ""); return
{title}
{sharedText &&
{sharedText}
} {}
1 ? { tabIndex: 0, role: "group", "aria-label": sharedText ? `${title} — ${sharedText}` : title } : {}}>
{showColHeaders &&
} {showColHeaders && colHeaders.map((h, i) =>
{h}
)} {showColHeaders &&
} {rows.map(r => [
{r.label}
, ...r.values.map((v, i) =>
{v !== null ? v : "—"}
)])}
{legend &&
{(Array.isArray(legend) ? legend : [legend]).map((line, i) =>
{line}
)}
}
; }; const buildSpeedTable = measurements => { if (measurements.length === 0) return null; const {shared, differing} = partitionWorkload(measurements); const sharedText = formatWorkloadParts(measurements[0] && measurements[0].workload, shared); const colHeaders = measurements.map(m => formatWorkloadParts(m && m.workload, differing)); const rows = SPEED_LABELS.map(tup => { const [key, label, unit, compute] = tup; const values = measurements.map(m => { const raw = compute ? compute(m) : m[key]; return fmt(raw, unit); }); return { label, values }; }); return { title: "Speed", sharedText, colHeaders, rows, colCount: measurements.length, legend: [`throughput per gpu = (input+output tokens)/elapsed/GPU`, `interactivity = 1000/TPOT(ms) (tokens/s/user)`] }; }; const buildAccuracyTable = accuracy => { if (!accuracy) return null; const rows = ACCURACY_LABELS.map(([key, label, unit]) => { const v = fmt(accuracy[key], unit); if (v === null) return null; return { label, values: [v] }; }).filter(r => r !== null); if (rows.length === 0) return null; return { title: "Accuracy", sharedText: null, colHeaders: [], rows, colCount: 1 }; }; const accuracy = effectiveAccuracy(entry, sel); const isEmpty = benchmarkIsEmpty(entry, accuracy); const measurements = !isEmpty ? normalizeSpeed(entry && entry.speed) : []; const accuracyTable = !isEmpty ? buildAccuracyTable(accuracy) : null; const speedTable = !isEmpty ? buildSpeedTable(measurements) : null; const hasBenchCmds = !isEmpty && buildBenchCommands(entry, sel) !== null; return
Benchmark
{!isEmpty && entry && entry.sglang_version &&
measured on sglang {entry.sglang_version}
} {hasBenchCmds && }
{isEmpty ?
Benchmark data pending for this combination — submit yours via the Playground's Submit ↗ button.
: <> {accuracyTable && renderBenchTable(accuracyTable)} {speedTable && renderBenchTable(speedTable)} {entry && entry.notes &&
{entry.notes}
} }
; }; const buildBenchCommands = (entry, sel) => { const bc = config.benchmarkCommands; if (!bc) return null; const acc = effectiveAccuracy(entry, sel); const accuracy = []; if (bc.accuracy) { for (const [key, label] of ACCURACY_LABELS) { if (acc[key] == null) continue; const tmpl = bc.accuracy[key]; const resolved = typeof tmpl === "string" ? tmpl : tmpl && tmpl[sel.variant] || null; if (resolved) accuracy.push({ key, label, template: resolved }); } } let speed = null; if (bc.speed && entry) { const ms = normalizeSpeed(entry.speed).filter(m => m && m.workload && m.workload.max_concurrency != null); const concurrencies = [...new Set(ms.map(m => m.workload.max_concurrency))].sort((a, b) => a - b); if (concurrencies.length) { speed = { template: bc.speed, concurrencies, workload: ms[0].workload, numPromptsOf: c => { const m = ms.find(x => x.workload.max_concurrency === c); if (m && m.workload.num_prompts != null) return m.workload.num_prompts; const tbl = bc.numPromptsByConc; if (tbl && tbl[c] != null) return tbl[c]; return Math.max(c * 2, 200); } }; } } if (accuracy.length === 0 && !speed) return null; return { accuracy, speed }; }; const buildHardwareGroups = () => { const supported = new Set(config.supportedHardware); const catalog = {}; for (const [vendor, list] of Object.entries(HARDWARE_CATALOG)) catalog[vendor] = [...list]; for (const hw of config.hardware || []) { const vendor = hw.vendor || "nvidia"; const list = catalog[vendor] || (catalog[vendor] = []); const entry = { id: hw.id, label: hw.label, vram: hw.vram }; const i = list.findIndex(x => x.id === hw.id); if (i >= 0) list[i] = entry; else list.push(entry); } const groups = []; for (const [vendor, list] of Object.entries(catalog)) { const items = list.filter(hw => supported.has(hw.id)).map(hw => ({ id: hw.id, label: hw.label, subtitle: hw.vram })); if (items.length) groups.push({ label: vendor.toUpperCase(), items }); } if (config.groupHardware === false) { return [{ label: null, items: groups.flatMap(group => group.items) }]; } return groups; }; const initialSelectionFromCells = () => { const first = (config.cells || [])[0]; const sel = Object.fromEntries(DIMENSIONS.map(d => [d, first ? first.match[d] : ""])); for (const spec of overlayDimSpecs) { const opts = spec.options || []; sel[spec.id] = (spec.default ?? (opts[0] && opts[0].id)) ?? ""; } if (!commandBuilder) return sel; return { ...sel, hw: commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || "", ...commandBuilder.defaultSelection || ({}) }; }; const normalizeBuilderSelection = parsed => { const out = { ...initialSelectionFromCells(), ...parsed }; if (!(config.supportedHardware || []).includes(out.hw)) { out.hw = commandBuilder.defaultSelection?.hw || config.supportedHardware?.[0] || ""; } for (const spec of overlayDimSpecs) { if (spec.kind === "number") { const value = Number.parseInt(out[spec.id], 10); out[spec.id] = Math.min(spec.max, Math.max(spec.min, Number.isFinite(value) ? value : Number(spec.default ?? spec.min))); continue; } const options = spec.options || []; if (!options.some(option => option.id === out[spec.id])) { out[spec.id] = (spec.default ?? options[0]?.id) ?? ""; } } for (const [key, bounds] of Object.entries(commandBuilder.resource?.limits || ({}))) { const fallback = Number((commandBuilder.defaultSelection?.[key] ?? bounds.min) ?? 1); const value = Number.parseInt(out[key], 10); out[key] = Math.min(bounds.max, Math.max(bounds.min, Number.isFinite(value) ? value : fallback)); } for (const key of ["tp_size", "ulysses_degree", "ring_degree"]) { const value = Number.parseInt(out[key], 10); out[key] = Number.isFinite(value) && value > 0 ? value : 1; } out.topology_mode = out.topology_mode === "manual" ? "manual" : "auto"; return out; }; const placeholderDefaults = schema => { const out = {}; for (const [k, v] of Object.entries(schema || ({}))) out[k] = v.default ?? ""; return out; }; const [isDark, setIsDark] = useState(false); useEffect(() => { const check = () => { const html = document.documentElement; setIsDark(html.classList.contains("dark") || html.getAttribute("data-theme") === "dark" || html.style.colorScheme === "dark"); }; check(); const observer = new MutationObserver(check); observer.observe(document.documentElement, { attributes: true, attributeFilter: ["class", "data-theme", "style"] }); return () => observer.disconnect(); }, []); const STORAGE_KEY = "sglang-deploy-env"; const [env, setEnv] = useState(() => placeholderDefaults(config.placeholders)); useEffect(() => { try { const raw = window.localStorage.getItem(STORAGE_KEY); if (raw) { const parsed = JSON.parse(raw); setEnv({ ...placeholderDefaults(config.placeholders), ...parsed }); } } catch {} }, []); const saveEnv = next => { setEnv(next); try { window.localStorage.setItem(STORAGE_KEY, JSON.stringify(next)); } catch {} }; const [sel, setSel] = useState(() => initialSelectionFromCells()); const [selectionHydrated, setSelectionHydrated] = useState(false); const INTERNAL_HASH_STATE_KEY = "__sglangDeployInternalHash"; const DEPLOYMENT_COMPONENT_ID = "deployment-configurator"; useEffect(() => { const hydrate = () => { const raw = window.location.hash.replace(/^#/, ""); if (!raw) return; const params = new URLSearchParams(raw); const initial = initialSelectionFromCells(); const parsed = { ...initial }; let touched = false; params.forEach((value, key) => { if ((key in parsed)) { parsed[key] = value; touched = true; } }); if (!touched) return; setSel(commandBuilder ? normalizeBuilderSelection(parsed) : validateSelection(config.cells, parsed)); const historyState = window.history.state; const isInternalHash = historyState && typeof historyState === "object" && historyState[INTERNAL_HASH_STATE_KEY] === `#${raw}`; if (isInternalHash) return; const el = document.getElementById(DEPLOYMENT_COMPONENT_ID); if (el) el.scrollIntoView({ behavior: "smooth", block: "start" }); }; hydrate(); setSelectionHydrated(true); window.addEventListener("hashchange", hydrate); return () => window.removeEventListener("hashchange", hydrate); }, []); useEffect(() => { if (!selectionHydrated) return; const target = "#" + new URLSearchParams(sel).toString(); if (window.location.hash !== target) { const historyState = window.history.state && typeof window.history.state === "object" ? window.history.state : {}; window.history.replaceState({ ...historyState, [INTERNAL_HASH_STATE_KEY]: target }, "", target); } window.dispatchEvent(new CustomEvent("sglang-deploy-sel", { detail: sel })); }, [sel, selectionHydrated]); const [modal, setModal] = useState(null); useEffect(() => { if (modal === null) return; const onKey = e => { if (e.key === "Escape") setModal(null); }; const prev = document.body.style.overflow; document.body.style.overflow = "hidden"; window.addEventListener("keydown", onKey); return () => { window.removeEventListener("keydown", onKey); document.body.style.overflow = prev; }; }, [modal]); const [copied, setCopied] = useState(false); const [curlCopied, setCurlCopied] = useState(false); const [envDraft, setEnvDraft] = useState(env); const [benchConc, setBenchConc] = useState(null); const [benchAcc, setBenchAcc] = useState(null); const [benchCopied, setBenchCopied] = useState(null); const configuredRunModes = typeof config.runModes === "function" ? config.runModes(sel) : config.runModes; const runModes = configuredRunModes || ["python", "docker"]; const [runMode, setRunMode] = useState(runModes[0]); const [builderScope, setBuilderScope] = useState("base"); const [builderServerSetting, setBuilderServerSetting] = useState(null); const [builderAdvanced, setBuilderAdvanced] = useState(false); const [serveExpanded, setServeExpanded] = useState(false); const [requestExpanded, setRequestExpanded] = useState(false); const [builderHeadAddress, setBuilderHeadAddress] = useState(""); const [builderNodeRank, setBuilderNodeRank] = useState(0); const [blockedNote, setBlockedNote] = useState(null); const flashBlockedNote = (dim, reason) => { const note = { dim, reason }; setBlockedNote(note); setTimeout(() => setBlockedNote(cur => cur === note ? null : cur), 4000); }; useEffect(() => { if (builderNodeRank >= Number(sel.nodes || 1)) setBuilderNodeRank(0); }, [sel.nodes, builderNodeRank]); const hasRunMode = runModes.includes(runMode); const fallbackRunMode = runModes[0]; const activeRunMode = hasRunMode ? runMode : fallbackRunMode; useEffect(() => { if (!hasRunMode) setRunMode(fallbackRunMode); }, [hasRunMode, fallbackRunMode]); useEffect(() => { if (modal === "env") setEnvDraft(env); }, [modal, env]); const [mambaRatio, setMambaRatio] = useState(null); useEffect(() => { const onRatio = e => setMambaRatio(e.detail && (e.detail.baseRatio || e.detail.ratio) || null); window.addEventListener("sglang-k3-mamba-ratio", onRatio); return () => window.removeEventListener("sglang-k3-mamba-ratio", onRatio); }, []); const s = makeStyles(isDark); const cell = commandBuilder ? commandBuilder.resolveDeployment(sel) : findCell(config.cells, sel); const builderMeta = cell && cell.builder || ({}); const verifyStatus = cellVerifyStatus(cell, sel); const cellWithRatio = (() => { if (!cell || !mambaRatio) return cell; if (cell.flags.some(f => f.startsWith("--mamba-full-memory-ratio") || f.startsWith("--max-mamba-cache-size"))) return cell; const flags = [...cell.flags]; const line = `--mamba-full-memory-ratio ${mambaRatio}`; const i = flags.findIndex(f => f.startsWith("--host")); if (i >= 0) flags.splice(i, 0, line); else flags.push(line); return { ...cell, flags }; })(); const commandEnv = commandBuilder ? { ...env, NODE_RANK: String(builderNodeRank), NODE0_IP: builderHeadAddress || "" } : env; const command = renderCommand(cellWithRatio, sel, commandEnv, activeRunMode); const effFlags = cell ? overlayCompose(cell.flags, sel) : []; const specAlgoFlag = effFlags.find(f => f.split(/[\s=]/)[0] === "--speculative-algorithm"); const specMrrFlag = effFlags.find(f => f.split(/[\s=]/)[0] === "--max-running-requests"); const mtpHint = !!specAlgoFlag && !specMrrFlag; const specPinnedHint = !!specAlgoFlag && !!specMrrFlag; const specMrrValue = specMrrFlag ? specMrrFlag.split(/[\s=]/).filter(Boolean)[1] || "" : ""; const SPEC_ALGO_LABEL = { EAGLE: "MTP", EAGLE3: "MTP", FROZEN_KV_MTP: "MTP", DSPARK: "DSpark", DFLASH: "DFlash", NGRAM: "N-gram", STANDALONE: "standalone draft" }; const specAlgoName = (() => { if (!specAlgoFlag) return "MTP"; const v = specAlgoFlag.split(/[\s=]/).filter(Boolean)[1] || ""; return SPEC_ALGO_LABEL[v.toUpperCase()] || v || "MTP"; })(); const renderWarn = text => { const out = []; const re = /\[([^\]]+)\]\(#([^)]+)\)/g; let last = 0; for (let m; m = re.exec(text); last = m.index + m[0].length) { if (m.index > last) out.push(text.slice(last, m.index)); const anchor = m[2]; out.push(); } if (last < text.length) out.push(text.slice(last)); return out; }; const modelName = resolveModelName(sel); const curlTemplate = typeof config.curl === "function" ? config.curl(sel, cell) : config.curl; const curlText = interpolate(curlTemplate || "", env, modelName); const hwGroups = buildHardwareGroups(); const benchEntry = benchmarks ? findBenchmark(benchmarks, sel) : null; const isOverlayDim = dim => overlayDimSpecs.some(d => d.id === dim); const findOption = (dim, value) => { const spec = [...matchDimSpecs, ...overlayDimSpecs].find(d => d.id === dim); return spec && (spec.options || []).find(o => o.id === value); }; const isEnabled = (dim, value) => { const opt = findOption(dim, value); if (opt && optionDisabled(opt, sel)) return false; if (commandBuilder && dim === "hw") return true; return isOverlayDim(dim) || isOptionAvailable(config.cells || [], sel, dim, value); }; const reseatHiddenPicks = next => { let out = next; for (const spec of [...matchDimSpecs, ...overlayDimSpecs]) { const opts = visibleOptions(spec, out).filter(o => !optionDisabled(o, out)); if (!opts.length) continue; if (!opts.some(o => o.id === out[spec.id])) { out = { ...out, [spec.id]: opts[0].id }; } } return out; }; const recommendedBuilderRecipe = hw => { const recipes = commandBuilder.resource?.verifiedRecipes || []; return recipes.find(entry => entry.hw === hw && entry.default) || recipes.find(entry => entry.hw === hw); }; const handleSelect = (dim, value) => { if (commandBuilder) { setSel(prev => { let next = { ...prev, [dim]: value }; if (dim === "hw") { const currentRecipe = recommendedBuilderRecipe(prev.hw); const nextRecipe = recommendedBuilderRecipe(value); const resourcesFollowPlatformDefault = !!currentRecipe && Number(prev.nodes) === Number(currentRecipe.nodes) && Number(prev.gpus_per_node) === Number(currentRecipe.gpus_per_node); next = { ...next, nodes: resourcesFollowPlatformDefault ? nextRecipe?.nodes ?? next.nodes : next.nodes, gpus_per_node: resourcesFollowPlatformDefault ? nextRecipe?.gpus_per_node ?? next.gpus_per_node : next.gpus_per_node, topology_mode: "auto", tp_size: resourcesFollowPlatformDefault ? nextRecipe?.tp_size ?? 1 : next.tp_size, ulysses_degree: resourcesFollowPlatformDefault ? nextRecipe?.ulysses_degree ?? 1 : next.ulysses_degree, ring_degree: resourcesFollowPlatformDefault ? nextRecipe?.ring_degree ?? 1 : next.ring_degree, placement: resourcesFollowPlatformDefault ? nextRecipe?.placement || "auto" : next.placement, encoder: resourcesFollowPlatformDefault ? nextRecipe?.encoder || "auto" : next.encoder }; } return reseatHiddenPicks(normalizeBuilderSelection(next)); }); return; } setSel(prev => reseatHiddenPicks(isOverlayDim(dim) ? { ...prev, [dim]: value } : snapToValidCell(config.cells, prev, dim, value))); }; const commitBuilderNumber = (event, currentValue, bounds, commit) => { const parsed = Number(event.currentTarget.value); if (!Number.isInteger(parsed)) { event.currentTarget.value = String(currentValue); return; } const value = Math.min(bounds.max, Math.max(bounds.min, parsed)); event.currentTarget.value = String(value); commit(value); }; const renderBuilderNumberInput = ({identity, value, min, max, label, onCommit}) => event.currentTarget.select()} onBlur={event => commitBuilderNumber(event, value, { min, max }, onCommit)} onKeyDown={event => { if (event.key === "Enter") event.currentTarget.blur(); }} />; const updateBuilderResource = (key, delta) => { if (!commandBuilder) return; const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); setSel(prev => { const value = Math.min(bounds.max, Math.max(bounds.min, Number(prev[key]) + delta)); return normalizeBuilderSelection({ ...prev, [key]: value, topology_mode: "auto" }); }); }; const setBuilderResource = (key, rawValue) => { if (!commandBuilder) return; const value = Number.parseInt(rawValue, 10); if (!Number.isFinite(value)) return; const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); setSel(prev => normalizeBuilderSelection({ ...prev, [key]: Math.min(bounds.max, Math.max(bounds.min, value)), topology_mode: "auto" })); }; const editBuilderTopology = (key, value) => { if (!commandBuilder) return; setSel(prev => normalizeBuilderSelection({ ...prev, topology_mode: "manual", [key]: Number.parseInt(value, 10) || 1 })); }; const handleCopy = () => { navigator.clipboard.writeText(command); setCopied(true); setTimeout(() => setCopied(false), 1200); }; const copyCurl = () => { navigator.clipboard.writeText(curlText); setCurlCopied(true); setTimeout(() => setCurlCopied(false), 1200); }; const copyBench = (key, text) => { navigator.clipboard.writeText(text); setBenchCopied(key); setTimeout(() => setBenchCopied(null), 1200); }; const placeholderGroups = (() => { const out = { command: [], curl: [] }; for (const [key, meta] of Object.entries(config.placeholders || ({}))) { (out[meta.target] || (out[meta.target] = [])).push({ key, ...meta }); } return out; })(); const renderButton = (item, dim, selectedId) => { const checked = selectedId === item.id; const disabled = !isEnabled(dim, item.id); return ; }; const renderFlatSection = (title, options, dim, selectedId) =>
{title}
{options.map(item => renderButton(item, dim, selectedId))}
; const maxHwCols = Math.max(...hwGroups.map(x => x.items.length)); if (commandBuilder) { const scopeLabel = { base: "Setup", serve: "Server", request: "Request" }; const scopedDims = scope => overlayDimSpecs.filter(dim => { if ((dim.scope || "base") !== scope) return false; if (dim.kind === "number") { return typeof dim.showWhen !== "function" || dim.showWhen(sel); } return rowVisible(dim, sel); }); const baseDims = scopedDims("base"); const serveDims = scopedDims("serve"); const requestDims = scopedDims("request"); const errors = builderMeta.errors || []; const warnings = builderMeta.warnings || []; const invalid = errors.length > 0; const totalGpus = Number(sel.nodes) * Number(sel.gpus_per_node); const topology = builderMeta.topology || ({}); const verification = builderMeta.verification || ({}); const scopeIsVerified = scope => scopedDims(scope).every(dim => { const option = (dim.options || []).find(entry => entry.id === sel[dim.id]); if (option && optionSoft(option, sel)) return false; const predicate = option?.verifiedWhen ?? dim.verifiedWhen; return typeof predicate === "function" ? !!predicate(sel) : predicate !== false; }); const serveStatus = invalid ? "error" : scopeIsVerified("serve") ? verification.serve || verifyStatus : "unverified"; const requestStatus = invalid ? "error" : scopeIsVerified("request") ? verification.request || verifyStatus : "unverified"; const statusText = status => ({ verified: "Verified", unverified: "Unverified", "in-progress": "Verification in progress", error: "Invalid configuration" })[status] || "Unverified"; const activeServerSetting = serveDims.find(dim => dim.id === builderServerSetting) || serveDims[0]; const selectedOption = dim => (dim.options || []).find(option => option.id === sel[dim.id]); const effectiveSetting = dim => builderMeta.resolvedSettings?.[dim.id] || selectedOption(dim)?.label || sel[dim.id] || "—"; const recommendedRecipe = recommendedBuilderRecipe(sel.hw); const recommendedInUse = !!recommendedRecipe && Number(sel.nodes) === recommendedRecipe.nodes && Number(sel.gpus_per_node) === recommendedRecipe.gpus_per_node && sel.topology_mode === "auto" && serveDims.every(dim => { const expected = recommendedRecipe[dim.id] ?? dim.default; if (dim.id === "attention") return sel.attention === "platform"; return expected === undefined || sel[dim.id] === expected || ["placement", "encoder"].includes(dim.id) && sel[dim.id] === "auto"; }); const restoreRecommendedRecipe = () => { if (!recommendedRecipe) return; setSel(prev => reseatHiddenPicks(normalizeBuilderSelection({ ...prev, ...Object.fromEntries(serveDims.map(dim => [dim.id, (recommendedRecipe[dim.id] ?? dim.default) ?? dim.options?.[0]?.id])), nodes: recommendedRecipe.nodes, gpus_per_node: recommendedRecipe.gpus_per_node, topology_mode: "auto", tp_size: recommendedRecipe.tp_size, ulysses_degree: recommendedRecipe.ulysses_degree, ring_degree: recommendedRecipe.ring_degree, placement: recommendedRecipe.placement || "auto", attention: "platform", precision: "native", encoder: recommendedRecipe.encoder || "auto", execution: recommendedRecipe.execution || "eager" }))); }; const renderBuilderChoice = (item, dim) => { const checked = sel[dim.id] === item.id; const disabled = !isEnabled(dim.id, item.id); const soft = !disabled && optionSoft(item, sel); const reason = disabled ? item.disableReason || "Not available for this configuration" : soft ? item.softReason || "Runs, but this combination is not a verified recipe yet." : ""; return ; }; const renderBuilderDimension = dim =>
{dim.title} {dim.description && {dim.description}}
5 ? "compact" : "normal"}> {visibleOptions(dim, sel).map(option => renderBuilderChoice(option, dim))}
{blockedNote && blockedNote.dim === dim.id &&

{blockedNote.reason}

}
; const renderStepper = (key, label, detail) => { const bounds = commandBuilder.resource?.limits?.[key] || ({ min: 1, max: 8 }); return
{label} {detail && {detail}}
{renderBuilderNumberInput({ identity: `${key}-${sel[key]}`, value: sel[key], min: bounds.min, max: bounds.max, label, onCommit: value => setBuilderResource(key, value) })}
; }; const renderBaseScope = () =>
{recommendedRecipe &&
{} {recommendedRecipe.unverified ? "Derived recipe" : "Verified recipe"} · {sel.hw.toUpperCase()} {[`${recommendedRecipe.nodes * recommendedRecipe.gpus_per_node} GPUs`, recommendedRecipe.tp_size > 1 && `TP ${recommendedRecipe.tp_size}`, `Ulysses ${recommendedRecipe.ulysses_degree}`, recommendedRecipe.ring_degree > 1 && `Ring ${recommendedRecipe.ring_degree}`, ({ resident: "Resident", fsdp: "FSDP", offload: "Layerwise offload" })[recommendedRecipe.placement]].filter(Boolean).join(" · ")}
{renderStatus(recommendedRecipe.unverified ? "unverified" : "verified")} {recommendedInUse ? In use : }
}
Hardware
{hwGroups.flatMap(group => group.items).map(item => { const selected = sel.hw === item.id; return ; })}
{}
Resources {builderMeta.topologySummary || "No valid topology"}
{renderStepper("nodes", "Nodes")} {renderStepper("gpus_per_node", "GPUs / node")}
{Number(sel.nodes) > 1 &&

{sel.nodes} nodes × {sel.gpus_per_node} {sel.hw.toUpperCase()} = {totalGpus} GPUs

} {builderAdvanced &&

Auto uses an exact verified recipe when one exists; manual values are allowed when the model constraints remain valid.

{[["tp_size", "Tensor parallel", [1, 2, 4, 8]], ["ulysses_degree", "Ulysses", [1, 2, 4, 8, 16]], ["ring_degree", "Ring", [1, 2, 4, 8]]].map(([key, label, values]) => )}
} {(errors.length > 0 || warnings.length > 0) &&
{(errors.length ? errors : warnings).map((message, index) =>

{message}

)}
}
{baseDims.map(renderBuilderDimension)}
; const renderSettingEditor = (dim, className = "", direct = false) => { if (!dim) return null; const options = visibleOptions(dim, sel); const currentOption = selectedOption(dim); return
{direct ? dim.title : `${dim.title} options`} {dim.description &&

{dim.description}

}
{dim.quality && {dim.quality}}
{dim.kind === "number" ?
{renderBuilderNumberInput({ identity: `${dim.id}-${sel[dim.id]}`, value: sel[dim.id], min: dim.min, max: dim.max, label: dim.title, onCommit: value => setSel(prev => ({ ...prev, [dim.id]: value })) })} {dim.unit || "outputs"}
:
{options.map(option => renderBuilderChoice(option, dim))}
} {blockedNote && blockedNote.dim === dim.id &&

{blockedNote.reason}

} {(currentOption?.description || dim.learnMore) &&
{currentOption?.description &&

{currentOption.description}

} {} {dim.learnMore && Learn more } {dim.docsHref && SGLang docs }
}
; }; const renderServerScope = () =>
{serveDims.map(dim => { const isActive = dim.id === activeServerSetting?.id; const option = selectedOption(dim); const recommended = typeof option?.recommendedWhen === "function" ? option.recommendedWhen(sel) : !!option?.recommended; return
{isActive && renderSettingEditor(dim, "sgd-builder-context--inline")}
; })}
{renderSettingEditor(activeServerSetting, "sgd-builder-context--rail")}
; const renderRequestScope = () =>
{requestDims.map(dim =>
{renderSettingEditor(dim, "", true)}
)}
; const renderScopeControls = () => { if (builderScope === "base") return renderBaseScope(); if (builderScope === "serve") return renderServerScope(); return renderRequestScope(); }; const renderStatus = status => ; const renderOutputCard = type => { const serve = type === "serve"; const text = serve ? command : curlText; const canExpand = text.split("\n").length > 9; const expanded = serve ? serveExpanded : requestExpanded; const setExpanded = serve ? setServeExpanded : setRequestExpanded; const status = serve ? serveStatus : requestStatus; const emphasized = builderScope === "base" || builderScope === type; return
{serve ? "1" : "2"}
{serve ? "Serve" : "Request"} {serve ? `${sel.hw.toUpperCase()} · ${activeRunMode === "docker" ? "Docker" : "Python"}` : "cURL"}
{renderStatus(status)}
{serve && runModes.length > 1 &&
{runModes.map(mode => )}
} {serve && Number(sel.nodes) > 1 &&
} 
{text}
{invalid &&
{errors[0]}
}
{canExpand && }
; }; return
{renderScopeControls()}
{builderScope !== "request" && renderOutputCard("serve")} {builderScope !== "serve" && renderOutputCard("request")}
{modal === "env" &&
setModal(null)}>
event.stopPropagation()}>
Command variables
{["command", "curl"].map(target => placeholderGroups[target].length > 0 &&
{target === "command" ? "Serve" : "Request"}
{placeholderGroups[target].map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: event.target.value })} />
)}
)}
}
; } return
{}
Hardware Platform
{hwGroups.map(g =>
{g.label &&
{g.label}
}
{g.items.map(item => renderButton(item, "hw", sel.hw))} {Array.from({ length: maxHwCols - g.items.length }).map((_, i) =>
)}
)}
{matchDimSpecs.filter(d => rowVisible(d, sel)).map(d =>
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
)} {overlayDimSpecs.filter(d => rowVisible(d, sel)).map(d =>
{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}
)} {}
Command:
{cell && cell.redirect ? cell.warn &&
⚠️ {renderWarn(cell.warn)}
: <>
{VERIFY_LABEL[verifyStatus]}
{runModes.map((mode, index) => setRunMode(mode)} onKeyDown={e => { if (e.key !== "Enter" && e.key !== " ") return; e.preventDefault(); setRunMode(mode); }} role="tab" tabIndex={0} aria-selected={activeRunMode === mode}> {mode === "docker" ? "Docker" : "Python"} )} 
{command}
{cell && cell.warn &&
⚠️ {renderWarn(cell.warn)}
} {mtpHint &&
⚠️ Speculative decoding ({specAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency.
} {specPinnedHint &&
ℹ️ Speculative decoding ({specAlgoName}) is on and this recipe pins --max-running-requests to {specMrrValue}. Adjust it to match your target concurrency — if you remove the flag, SGLang falls back to 48.
} }
{} {benchmarks && cell && renderBenchmarkCard(benchEntry)} {} {config.showPlaygroundLink !== false &&
Need to go beyond the verified matrix?
} {} {modal === "curl" &&
setModal(null)}>
e.stopPropagation()}>
cURL example
Model: {modelName || "(unresolved)"} 
{curlText}

Edit CURL_HOST / CURL_PORT in the Env panel.

} {} {modal === "env" &&
setModal(null)}>
e.stopPropagation()}>
Env / placeholder values
{placeholderGroups.curl.length > 0 &&
cURL placeholders
{placeholderGroups.curl.map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: e.target.value })} />
)}
} {placeholderGroups.command.length > 0 &&
Command placeholders
{placeholderGroups.command.map(({key, label}) =>
setEnvDraft({ ...envDraft, [key]: e.target.value })} />
)}
}

Values persist in localStorage and are reused the next time you visit any cookbook.

} {} {modal === "bench" && benchEntry && (() => { const bc = buildBenchCommands(benchEntry, sel); if (!bc) return null; const selSummary = [sel.hw && sel.hw.toUpperCase(), sel.variant, sel.quant && sel.quant.toUpperCase(), sel.strategy, sel.kvDsaPair, sel.nodes].filter(part => part !== undefined && part !== null && part !== "").join(" · "); let selConc = null; let speedCmd = null; if (bc.speed) { selConc = bc.speed.concurrencies.includes(benchConc) ? benchConc : bc.speed.concurrencies[0]; const w = bc.speed.workload; speedCmd = interpolate(bc.speed.template, { ...env, DATASET: w.dataset, ISL: w.isl, OSL: w.osl, MAX_CONCURRENCY: selConc, NUM_PROMPTS: bc.speed.numPromptsOf(selConc) }, modelName); } let selAcc = null; let accCmd = null; if (bc.accuracy.length > 0) { selAcc = bc.accuracy.find(a => a.key === benchAcc) || bc.accuracy[0]; accCmd = interpolate(selAcc.template, env, modelName); } return
setModal(null)}>
e.stopPropagation()}>
Benchmark commands

For {selSummary}. Start the server with the Deploy command above, then run these against it.

{selAcc &&
Accuracy
{bc.accuracy.length > 1 &&
benchmark: {bc.accuracy.map(a => )}
}
{selAcc.label}
{accCmd}
{bc.accuracy.length > 1 &&

Switch the benchmark chip to see each eval's command.

}
} {bc.speed &&
Speed
{bc.speed.concurrencies.length > 1 &&
max-concurrency: {bc.speed.concurrencies.map(c => )}
}
max-concurrency = {selConc}
{speedCmd}

One command — switch the concurrency chip (or edit --max-concurrency) to reproduce each Speed column.

}

Edit CURL_HOST / CURL_PORT in the Env panel.

; })()}
; }; export const DiffusionModelTags = ({tags = []}) => { const normalizedTags = Array.isArray(tags) ? tags : [tags]; return
{normalizedTags.map(tag => {tag} )}
; }; ## 1. Quick start Use Linux with NVIDIA CUDA and a checkout containing this implementation; it is not yet in a released package or unmodified release image. Authenticate an account with checkpoint access using `hf auth login`, or supply `HF_TOKEN` to the server. From the repo root: ```bash Install theme={null} uv pip install -e "python[diffusion]" --prerelease=allow ``` The picker uses the distilled two-step checkpoint. Four-GB300 whole-tile decode is the output-preserving choice measured on multi-tile clips; one GPU suffices for a functional check when it has enough memory. The picker also includes explicitly **extrapolated** Hopper and RTX 5090 starting points, not measured speed or memory guarantees. HTTP verification is scoped separately from offline measurements. See the [installation guide](/docs/sglang-diffusion/installation) for platform setup. The request uploads a local video and returns a job `id`. Poll `GET /v1/videos/{id}`, then download `GET /v1/videos/{id}/content` when completed. Use the shared [polling and download example](/cookbook/diffusion/Kandinsky/Kandinsky6#5-download-a-completed-video). For JSON requests, `video_path` refers to a file on the **server**, not the client. ## 2. Model capabilities Kandinsky 6 VSR upscales a source video without a prompt. It encodes the source with a causal KVAE, denoises overlapping tiles and blends their decoded outputs. Unlike the [generation model](/cookbook/diffusion/Kandinsky/Kandinsky6), it retains the source soundtrack rather than generating audio. Audio is converted to mono 44.1 kHz and trimmed to the processed video duration. Use `kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` for x2, x4 or x2.25 upscaling of up to 121 frames. Its checkpoint-selected PiFlow scheduler uses two DiT calls per tile; changing `num_inference_steps` does not increase quality or the number of evaluations. The native path uses dense attention, not the checkpoint's requested NABLA sparse attention, so this is not a claim of numerical parity with the sparse reference. The tested non-distilled `Kandinsky-6.0-VSR-5s-Diffusers` revision `d6ddfabc62b3920b6da6487e4f444d37c591d5f4` declares a flow-Euler scheduler but contains a 640-wide output head instead of the required 64. It is not supported as-is. Do not change the scheduler to bypass this error; use the distilled model. ## 3. Offline usage No prompt is required. A single-GPU example: ```bash Generate theme={null} sglang generate \ --model-path kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers \ --video-path /path/to/input.mp4 \ --sr-resolution-scale 2 \ --num-inference-steps 2 \ --seed 42 --save-output --output-path ./sr_outputs ``` ```python Generate theme={null} from sglang.multimodal_gen import DiffGenerator generator = DiffGenerator.from_pretrained( model_path="kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers", num_gpus=1, ) try: result = generator.generate(sampling_params_kwargs={ "video_path": "/path/to/input.mp4", "sr_resolution_scale": 2, "num_inference_steps": 2, "seed": 42, "output_path": "sr_outputs", "save_output": True, }) print(result.output_file_path) finally: generator.shutdown() ``` ### Request parameters * `sr_resolution_scale`: `2`, `4` or `2.25` (default). The x2.25 path first enlarges pixels x1.125, then processes x2 tiles. * `sr_tiles_batch_size`: default `1`. More tiles per DiT call need more activation memory. Each chunk is seeded by its first tile index, so changing this value changes the noise and output. It is **not a lossless batching optimization**. * `sr_tile_min_overlap`: default `0.2`. Changing overlap changes tile geometry and blending, not just performance. * `sr_target_resolution`: optional `hd`, `fullhd`, `2k` or `WxH`; only downscales the generated result. `sr_target_resize_mode` is `fit` (default) or `exact`. * `seed`: default `42`. Compare identical seeds, tile batching, source frames and attention policy when evaluating a decode optimization. The input must fit at least one tile. A tested landscape source is 384 x 256. The default filename is `_srx_.mp4`; override it with `--output-file-name`. ## 4. Decode speed, memory and numerics These are **VAE decode choices**, independent of DiT TP/Ulysses/Ring:
DecodeHot E2EPeak reservedOutput versus serial
Serial45.26 s47.55 GiBReference
Whole-tile parallel21.73 s47.55 GiBRGB and audio exact in this test
Spatial shard18.01 s24.37 GiBRGB differs; audio exact
Setup: four GB300 GPUs (277.5 GiB each, NV18 links), host RAM 955.7 GiB with an 860 GiB cgroup limit; BF16/FA, Ulysses4, resident weights, no compile or startup warmup. PyTorch 2.14.1+cu130, Transformers 5.17.0, Diffusers 0.37.0. Distilled checkpoint revision `3e3e10bcd23659eeeb23cac740258ee4fc14aa7d`; seed 42, tile batch 1, two steps per tile. A generated kitchen clip, 768 x 512 x 121 frames at 24 fps, was upscaled x2 to 1536 x 1024. Times are medians of three hot requests after one initial request, including MP4 saving and excluding model loading. Memory is PyTorch request peak reserved on the output rank, not loading, NVML or the maximum across ranks. Host memory is not claimed unchanged. **Whole-tile parallel** assigns complete existing tiles to ranks and preserves their temporal caches and blend order. It matched serial RGB and raw audio exactly on the tested workloads **within the same topology**. Select `--vae-config.use-parallel-tiling true`. A single tile falls back to serial. Communication can outweigh savings for small clips or slow interconnects. **Spatial shard is not bitwise lossless.** It splits each tile's height, using halo exchanges and the original temporal cache. Changed convolution shapes change rounding. In the experiment above, all-frame RGB PSNR was **57.51 dB**, minimum SSIM over the first/middle/last frames **0.99921**, mean absolute difference **0.1132/255**, and maximum difference **4/255**. Audio was exact. These measurements are not a guarantee for other videos or perceptual-quality certification. Select it only after validating the output for your workload: ```bash Spatial decode flags theme={null} --vae-config.use-parallel-decode true --vae-config.parallel-decode-mode spatial_shard ``` Both modes are opt-in runtime features. The picker selects whole-tile parallel for its multi-tile recipe; it does not change runtime defaults. If both flags are enabled, spatial sharding takes precedence. One GPU or a tile too short to split retains replicated decoding. Neither mode changes the tile geometry. ### RTX PRO 6000 over PCIe | GPUs / placement | Decode | Hot E2E | Sampled GPU peak | | - | - | - | - | | 1 / resident | Serial | 134.09 s | 48.91 GiB | | 2 / resident, Ulysses2 | Serial | 122.11 s | 48.81 GiB | | 2 / resident, Ulysses2 | Whole-tile | 83.04 s | 48.81 GiB | | 2 / resident, Ulysses2 | Spatial | 73.62 s | 37.45 GiB | | 1 / all-layerwise offload | Serial | 133.60 s | 37.43 GiB | Setup: two RTX PRO 6000 Blackwell Server Edition GPUs (96GB nominal), PCIe/SYS across CPU sockets, **no NVLink**; 499.2 GiB host RAM, no cgroup cap. Driver 595.84, the same PyTorch/Transformers/Diffusers versions and checkpoint as above, but **SDPA**, not FA. A separate generated 768 x 512 x 121-frame clip at 24 fps was upscaled x2 with seed 42, tile batch 1 and two steps per tile. Each row uses one initial request and three hot requests, no compile/startup warmup. E2E includes MP4 saving and raw-frame return, excludes model loading. GPU memory is the maximum **per-GPU NVML sample across ranks**, sampled every 0.5 seconds during requests, not the sum or PyTorch reserved memory; brief peaks may be missed. These are within-machine comparisons, not GB300-versus-RTX speed ratios. Ulysses2 alone reduced latency by only 8.9%. Whole-tile decode reduced it by 38.1% versus one GPU, with **exact RGB/audio** in this test. Spatial reached 45.1% lower latency, but RGB differed: PSNR **57.54 dB**, minimum sampled SSIM **0.99919**, maximum difference **4/255**; audio remained exact. All five configurations reproduced their own outputs across repeated requests. Layerwise offload reduced memory with essentially unchanged latency here; this does not make offload free on other workloads or slower host links. A separate **memory-budget probe**, still on RTX PRO 6000, completed the full clip twice with all-layerwise offload and a **30 GiB PyTorch allocator cap**. Sampled whole-GPU usage peaked at **31.03 GiB**, and RGB/audio matched the resident baseline exactly. The cap excludes CUDA allocations outside PyTorch; it is a test setting, not a deployment recommendation. This supports trying the 32GB recipe, but is **not a 5090 speed, memory-fit or quality measurement**. The complete HTTP regression also passed on one-GPU resident serial, Ulysses2 resident whole-tile and one-GPU layerwise serial with the allocator cap: six requests each covering JSON, uploads, x2/x4/x2.25, tile batch 2, audio/silent inputs and repeated-request equality. Arbitrary videos still need their own validation. ### Small workloads do not necessarily scale On GB300, a 384 x 256 x 33-frame source upscaled x2 (one tile, serial decode, otherwise the same BF16/FA/seed/step policy) took **1.77 s on one GPU, 1.72 s on Ulysses2 and 1.74 s on Ulysses4**. These are three-hot-request medians including saving. The roughly 2% difference is not evidence of a useful speedup; prefer one GPU for this workload. More tiles, not just more GPUs, create work that whole-tile decoding can distribute. ## 5. Memory and parallelism ### Hardware and GPU counts | Hardware | Starting point | Evidence boundary | | - | - | - | | GB300 | Resident, FA; Ulysses4 + whole-tile for multi-tile clips | Measured offline; smaller 1/2/4-GPU workloads also tested | | RTX PRO 6000 96GB | Resident, SDPA; compare 1 GPU with Ulysses2 | PCIe measurements, not a substitute for 5090 memory validation | | H100 80GB / H200 141GB | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated from native CUDA support; no Hopper SR measurement | | B200 | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated SR recipe; generation-model tests do not validate SR | | RTX 5090 32GB | SDPA, layerwise offload, tile batch 1; short 384 x 256 input first | Extrapolated from the RTX memory-budget probe; no 5090 execution claim | Only compare GPU counts on the **same input, scale, tile batching and backend**. For small single-tile clips, one GPU avoids communication overhead. For larger multi-tile clips, try Ulysses2/4 with whole-tile decode; extra GPUs do not divide replicated KVAE/latent-upscaler weights. Spatial decode can reduce activation memory, but changes rounding. Offload needs sufficient host RAM as well as VRAM. Do not extrapolate NVLink scaling to a PCIe system, especially across CPU sockets. Eight-GPU FA configurations are selectable as **Ulysses4 x Ring2**, not Ulysses8: 28 heads are not divisible by 8. This is an unmeasured topology, not a speedup recommendation. The current RTX SDPA path has no Ring support, so the picker rejects that combination. Start with fewer GPUs before trying an eight-GPU run. ### Placement and topology DiT Ulysses, TP, Ring and TP x SP are supported. For example, use `--num-gpus 4 --tp-size 1 --ulysses-degree 4` for Ulysses4, or `--num-gpus 4 --tp-size 2 --ulysses-degree 2` for TP2 x Ulysses2. TP times Ulysses must divide the checkpoint's 28 attention heads. **TP and Ring can change output numerics independently of decode mode**; whole-tile decode being exact does not make a TP/Ring topology change lossless. The decode group includes TP/SP ranks within each DP replica, not different requests. KVAE encoding, latent upscaling and tile scheduling remain replicated. For short two-step clips, extra GPUs can add communication without a speedup. Keep components resident when memory permits. For less GPU weight memory, use `--layerwise-offload-components all`, or select `vae latent_upscaler` instead. Temporal caches and non-block weights remain on GPU. Offload is compatible with both decode modes, but consumes host RAM and adds transfers for tiles and temporal segments. The assembled output also occupies host memory until saved. The **Server** tab also offers whole-component offload with `--cpu-offload-components`, selectable offload components, and DiT prefetch and resident-layer counts when streaming the DiT. Increasing those counts consumes additional VRAM. Partial offload recipes, non-default layer counts and FSDP (`--use-fsdp-inference true`) are **unverified options**, not measured improvements. FSDP requires multiple GPUs; SDPA cannot be combined with Ring. The default backend is FA on datacenter Blackwell/Hopper and SDPA on RTX. Hardware starting points do not automatically adapt to every uploaded video's memory demand. Request scale and tile batching do not require restarting the server. ## 6. Limits * **Input:** at most 121 frames. Sources near 24 fps retain their frames; faster videos are subsampled, slower videos retain their native rate. This can change the source timing; SR is not a bit-preserving video/audio transcode. * **Attention:** this native path is dense. Do not compare it with a NABLA sparse reference and attribute every difference to parallel decoding. The latent upscaler uses SDPA motion attention when `natten` is unavailable. * **Deployment:** monolithic only. Disaggregated roles, `torch.compile`, LoRA, direct latent inputs and KVAE-bridge chaining are not supported. * **Loading:** component config and weight mismatches fail explicitly. For a renamed local checkpoint, pass `--model-id Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` to retain SR routing. See the [compatibility matrix](/docs/sglang-diffusion/compatibility_matrix) for other model families. The opt-in server regression is `python/sglang/multimodal_gen/test/server/test_server_kandinsky6_sr.py`; it covers uploads, JSON, all three scales, source audio and repeated requests, not official pipeline parity. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.