# PaddleOCR-VL Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/PaddleOCR-VL Deploy PaddleOCR-VL 1.6 / 1.5 / 0.9B with SGLang — Baidu's 0.9B NaViT + ERNIE-4.5 document-parsing VLM for OCR, tables, formulas and charts in 109 languages, on a single H100, H200 or B200. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick a release and your hardware to generate the launch command. The model is 0.9B and single-GPU, so there is one serving recipe per platform; the axis that actually moves cost is **Page Resolution**, which caps how many image tokens one page is worth. ## Playground Use the Playground to layer tensor parallelism on top of the selected deployment cell. At this size TP is a latency knob, not a capacity one — the weights fit on one GPU. ## 1. Model Introduction **PaddleOCR-VL** is Baidu's compact document-parsing vision-language model: a NaViT-style dynamic-resolution SigLIP vision encoder feeding an **ERNIE-4.5-0.3B** language backbone, 0.9B parameters in total, released under **Apache 2.0**. It recognizes text, tables, formulas, charts and seals across **109 languages**, and is small enough that a single GPU serves it comfortably. SGLang serves **this model**, which is the recognition stage of PaddleOCR's document-parsing pipeline — not the pipeline itself. Layout and region detection, cropping, reading order and the assembly of markdown or JSON stay in PaddleOCR, which calls a model endpoint once per region. That split is why the model is prompt-conditioned rather than instruction-following: the caller decides whether a crop is a table, a formula or a chart (§3.1), and SGLang is the backend that answers. All three releases share an identical `config.json` (same `PaddleOCRVLForConditionalGeneration` architecture, same tower and backbone dimensions), so one SGLang recipe serves every variant and only the model path changes.
Variant Total params Use
PaddleOCR-VL-1.6 0.9B Latest. Best tables, Chinese characters and seals; drop-in for 1.5.
PaddleOCR-VL-1.5 0.9B Previous generation; pin it if you have calibrated against its output.
PaddleOCR-VL 0.9B The original 0.9B release.
**Recommended generation:** greedy decoding (`temperature=0`) with a per-page `max_tokens` budget — the model card uses 512 for a single region and the reference server allows more for a full page. These are informational; do not hardcode them in library code. **Resources:** [Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) · [PaddleOCR on GitHub](https://github.com/PaddlePaddle/PaddleOCR) ## 2. Configuration Tips * **Page resolution is the main cost knob.** The vision tower and the prefill both scale with the patch count of a page. `max_pixels` is expressed in 28x28 units (patch size 14 with a 2x2 merge), so `max_pixels / 784` is the image-token budget per page. The checkpoint's own default is 1280 tokens; the **Page Resolution** selector in the Deploy panel emits the corresponding `--mm-process-config` value. Lower it for clean born-digital PDFs, raise it for dense scans and small print. * **Prompt selects the task.** PaddleOCR-VL is prompt-conditioned rather than instruction-following — use the exact task strings in §3.1. A free-form question will not behave like a chat model. * **Leave `--trust-remote-code` off.** The checkpoints ship their own `configuration_paddleocr_vl.py` / `processing_paddleocr_vl.py`, but `transformers` 5.12 supports `paddleocr_vl` natively — and the bundled remote image processor is the slower of the two implementations (measured 87.4 ms vs 39.1 ms per 1080p page). Passing the flag pins SGLang to the remote copy. Serving without it produced byte-identical OCR output on every page we checked and about 5% more requests per second at 32-way concurrency. * **Preprocessing is parallelized for you.** A full-resolution page costs tens of milliseconds of CPU to resize, normalize and patchify, which caps throughput long before the GPU saturates, so this model runs the image processor across several workers by default. `--mm-processor-worker-num` overrides the count; raising it past the default did not help in our measurements. * **Keep the radix cache on for repeated pages.** Unlike whole-document batch OCR over unique scans, a workload that re-asks about the same page (different task prompts on one image) reuses the image prefix. Add `--disable-radix-cache` only if every request carries a different page. * **The saturated-throughput flags earn their place.** A page is \~2700 tokens, so the default 8192-token prefill budget packs only three of them into a forward. Raising it to 16384 and letting decode ride along in the same batch (`--enable-mixed-chunk`, `--num-continuous-decode-steps 2`) measured +11% requests per second at 32-way concurrency and cut queued TTFT by 23%, with single-stream latency unchanged. Measured on an H200; on a smaller card lower `--chunked-prefill-size` until it fits. * **Prefill CUDA graph is on for this model.** SGLang normally switches the breakable prefill graph off for every multimodal architecture; PaddleOCR-VL is allowlisted back in, which is worth 16.1 ms → 11.5 ms of single-stream TTFT on text-only prompts. Image-carrying batches are rejected at graph replay and run eager, so this helps mixed and text traffic, not pure page parsing. No flag needed. * **Tensor parallelism is optional.** The weights are under 2 GB in BF16; TP>1 only shortens the vision-encoder and prefill critical path, at the cost of a collective per layer. Measure before adopting it. * **Context length.** The backbone advertises 131072 positions, but a parsed page rarely needs more than a few thousand tokens. The recipe pins `--context-length 16384` so the KV pool stays small and concurrency stays high; raise it only if you batch many pages into one request. ### Measured on one H200 One 1080p page (\~2700 image tokens) in, 128 tokens out, prefix cache disabled, median TTFT:
Configuration TTFT, 1 stream req/s at 32 concurrent
With --trust-remote-code (remote image processor) 219 ms 10.9
Recipe above (native image processor) 114 ms 11.3
Throughput at saturation is bound by the vision tower, which runs full attention over every patch of the page — so the **Page Resolution** selector is the lever that moves it, not tensor parallelism. ## 3. Advanced Usage ### 3.1 Task prompts PaddleOCR-VL exposes its capabilities through a small set of fixed prompts. Send the prompt as the text part and the page as the image part of the same user turn.
Prompt Task
OCR: Plain text recognition.
Table Recognition: Table structure and cell contents.
Formula Recognition: Mathematical expressions.
Chart Recognition: Chart contents.
Spotting: Text with locations. Benefits from the high-detail resolution setting.
Seal Recognition: Seals and stamps (1.6).
Structured tasks answer in the model's own markup, not HTML: `Table Recognition:` returns OTSL-style cell tokens (`` per cell, `` per row), so a caller that wants HTML or markdown converts them itself. ```text Output theme={null} MethodsRPFFPSSegLink [26]70.086.077.08.9PixelLink [4]73.283.077.8-TextSnake [18]73.983.278.31.1 ... (one per cell, one per row, to the end of the table) ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.6", messages=[ { "role": "user", "content": [ {"type": "text", "text": "OCR:"}, { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.9/doc/imgs_en/img_12.jpg" }, }, ], } ], max_tokens=2048, ) print(response.choices[0].message.content) ``` ```text Output theme={null} ACKNOWLEDGEMENTS We would like to thank all the designers and contributors who have been involved in the production of this book; their contributions have been indispensable to its creation. We would also like to express our gratitude to all the producers for their invaluable opinions and assistance throughout this project. And to the many others whose names are not credited but have made specific input in this book, we thank you for your continuous support. ``` ### 3.2 Parsing a multi-page document The model parses one page per request. Render each page to an image, then fan the pages out concurrently — SGLang batches the vision encoders of in-flight requests into a single forward, so concurrency is what keeps the GPU busy on a model this small. ```python Example theme={null} import base64 from concurrent.futures import ThreadPoolExecutor import pymupdf from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") def render(page, dpi=200): pixmap = page.get_pixmap(dpi=dpi) return base64.b64encode(pixmap.tobytes("png")).decode("ascii") def parse(page_png_b64): response = client.chat.completions.create( model="PaddlePaddle/PaddleOCR-VL-1.6", messages=[ { "role": "user", "content": [ {"type": "text", "text": "OCR:"}, { "type": "image_url", "image_url": { "url": f"data:image/png;base64,{page_png_b64}" }, }, ], } ], max_tokens=2048, ) return response.choices[0].message.content document = pymupdf.open("your_document.pdf") pages = [render(page) for page in document] with ThreadPoolExecutor(max_workers=16) as pool: for index, text in enumerate(pool.map(parse, pages)): print(f"--- page {index + 1} ---") print(text) ``` ```text Output theme={null} --- page 1 --- (a) Total-Text (b) Total-Text (c) CTW1500 (d) CTW1500 Figure 8. Visual experimental results. The blue contours are boundary proposals, and the green contours are final detection boundaries. Table 6. Experimental results on CTW-1500. Methods Ext R P F FPS TextSnake [18] Syn 85.3 67.9 75.6 - ... (page continues) --- page 2 --- ... (one block per page, in page order) ``` # Unlimited-OCR Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/Unlimited-OCR Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention. ## Deployment Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/29186/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image. ## Playground Use the Playground to adjust tensor parallelism on top of the selected deployment cell. ## 1. Model Introduction [Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode. The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`. **Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186) ## 2. Configuration Tips * **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required. * **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`. * **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text. * **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command. * **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`. * **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`. ## 3. Advanced Usage ### 3.1 OCR request ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="baidu/Unlimited-OCR", messages=[ { "role": "user", "content": [ {"type": "text", "text": "document parsing."}, { "type": "image_url", "image_url": { "url": "https://example.com/your_document.png" }, }, ], } ], max_tokens=2048, temperature=0, extra_body={"images_config": {"image_mode": "gundam"}}, ) print(response.choices[0].message.content) ``` ### 3.2 Choosing an image mode Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.
Mode Use Multiple images
tiny Lowest prefill cost. Yes
small Lightweight OCR requests. Yes
base Balanced quality and cost. Yes
large Higher resolution single-image OCR. No
gundam Default high-detail document parsing mode. No
# Ornith-1.0 Source: https://docs.sglang.io/cookbook/autoregressive/DeepReinforce/Ornith-1.0 Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants. ## Deployment Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `...` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground). ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow "sglang>=0.5.9" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`. ## Playground The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end. For Ornith-1.0 the knobs are the reasoning and tool-call parsers: * **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `...` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`. * **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`. ## 1. Model Introduction [Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks. **Key Features:** * **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results. * **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts. * **Reasoning model behavior**: assistant responses begin with a `...` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`. * **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground). * **Long context**: model-card recipes use `--context-length 262144`. * **MIT license**: the Hugging Face repos are released under MIT. **Available Models:**
Model Format Deploy Panel Notes
deepreinforce-ai/Ornith-1.0-397B BF16 H200 only Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.
deepreinforce-ai/Ornith-1.0-397B-FP8 FP8 H100 / H200 FP8 repo in the collection; the deploy command uses this repo id with TP=8.
deepreinforce-ai/Ornith-1.0-35B BF16 H100 / H200 35B MoE checkpoint; the deploy command uses TP=2.
deepreinforce-ai/Ornith-1.0-35B-FP8 FP8 H100 / H200 FP8 repo in the collection; the deploy command uses this repo id with TP=2.
deepreinforce-ai/Ornith-1.0-9B BF16 H100 / H200 Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.
deepreinforce-ai/Ornith-1.0-35B-GGUF GGUF No Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.
deepreinforce-ai/Ornith-1.0-9B-GGUF GGUF No Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.
**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE) **Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html) ## 2. Configuration Tips * **Reasoning parser**: Ornith responses include `...`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`. * **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `` blocks are returned as OpenAI-compatible tool calls. * **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom. * **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget. * **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters. * **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off. ## 3. Usage Examples ### 3.1 Basic Chat Completion `message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `...` trace stays inline in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="Ornith-1.0-9B", messages=[ {"role": "user", "content": "Write a compact Python function is_prime(n)."} ], temperature=0.6, top_p=0.95, max_tokens=1024, extra_body={"top_k": 20}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ### 3.2 Tool Calling Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "run_tests", "description": "Run the project's test suite.", "parameters": { "type": "object", "properties": { "target": {"type": "string", "description": "Test target or command"} }, "required": ["target"], }, }, }] response = client.chat.completions.create( model="Ornith-1.0-9B", messages=[{"role": "user", "content": "Run the unit tests for the parser module."}], tools=tools, tool_choice="auto", temperature=0.6, top_p=0.95, max_tokens=2048, ) print(response.choices[0].message.tool_calls) ``` # DeepSeek-Math-V2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2 ## 1. Model Introduction [DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute. **Key Features:** * **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024 * **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy * **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024 * **Large MoE Model**: \~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB) **Available Models:** * **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights **License:** To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details. ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model. ### 3.2 Configuration Tips **Hardware Requirements:** * **B200 (183GB)**: BF16 tp=8 * **B300 (275GB)**: BF16 tp=8 **DP Attention:** * Enable DP attention for high-throughput scenarios * The `--dp` value commonly matches the `--tp` value * Trade-off: Higher throughput at the cost of slightly increased latency ## 4. Model Invocation ### 4.1 Deployment Command Deploy the model using the command generated above. Example for B200: ```shell Command theme={null} sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \ --tp 8 \ --ep 8 \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 Mathematical Reasoning DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning. **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Mathematical reasoning problem response = client.chat.completions.create( model="deepseek-ai/DeepSeek-Math-V2", messages=[ {"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"} ], max_tokens=4096, stream=True ) # Process the stream thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2. This is a classic formula for the sum of the first n natural numbers. We can prove by induction. Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds. Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1 )((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n. ... =============== Content ================= We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below. --- ### 1. Proof by mathematical induction **Base case (\(n=1\))**: \[ 1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1, \] so the formula holds for \(n=1\). **Inductive hypothesis:** Assume that for some positive integer \(k\) the formula is true, i.e. \[ 1+2+\dots+k = \frac{k(k+1)}{2}. \] **Inductive step (\(k \to k+1\))**: Consider the sum up to \(k+1\): \[ \begin{aligned} 1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt] &= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt] &= (k+1)\left(\frac{k}{2}+1\right)\\[4pt] &= (k+1)\frac{k+2}{2}\\[4pt] &= \frac{(k+1)(k+2)}{2}\\[4pt] &= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}. \end{aligned} \] Thus the formula also holds for \(n=k+1\). By the principle of mathematical induction, \[ 1+2+3+\dots+n = \frac{n(n+1)}{2} \] for every positive integer \(n\). --- ### 2. Proof by pairing (Gauss’s trick) Let \[ S = 1 + 2 + 3 + \dots + n. \] Write the same sum in reverse order: \[ S = n + (n-1) + (n-2) + \dots + 1. \] Add the two equalities term‑by‑term: \[ \begin{aligned} 2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\ &= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\ &= n\,(n+1). \end{aligned} \] Therefore \[ S = \frac{n(n+1)}{2}. \] Both proofs are rigorous and show that the formula holds for all positive integers \(n\). ``` ### 4.3 Competition-Level Problems **Example: IMO-style Problem:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # IMO-style problem response = client.chat.completions.create( model="deepseek-ai/DeepSeek-Math-V2", messages=[ {"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."} ], max_tokens=8192, stream=True ) # Process the stream thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to prove that for positive real numbers a,b,c with abc = 1, we have: \[ (a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1. \] We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite: ... =============== Content ================= We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove \[ \Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 . \] --- ### 1. A convenient substitution Because \(abc=1\), we can write \[ a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x} \] with positive numbers \(x,y,z\). (For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).) --- ### 2. Rewriting the factors \[ \begin{aligned} a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm] b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm] c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}. \end{aligned} \] Hence the product becomes \[ P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr) =\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}. \] --- ### 3. Reducing to a known inequality We have to show \(P\le1\), i.e. \[ (x+z-y)(x+y-z)(y+z-x)\le xyz . \tag{1} \] Set \[ p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz . \] Notice that \[ x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x . \] Therefore \[ \begin{aligned} (x+z-y)(x+y-z)(y+z-x) &=(p-2x)(p-2y)(p-2z)\\ &=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\ &=-p^{3}+4pq-8r . \end{aligned} \] Inequality (1) is thus equivalent to \[ -p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r . \tag{2} \] --- ### 4. Applying Schur’s inequality Schur’s inequality of third degree states that for any non‑negative \(x,y,z\) \[ p^{3}+9r\ge 4pq . \] Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2). Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds. Consequently (1) is true, and we obtain \(P\le1\). --- ### 5. Equality case Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\). Then \(a=b=c=1\), and indeed the product equals \(1\). --- Thus for all positive \(a,b,c\) with \(abc=1\), \[ \Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 . \] ∎ ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Benchmark **Benchmark Command:** ```shell Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --port 30000 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.975 Invalid: 0.000 Latency: 34.358 s Output throughput: 540.162 token/s ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x, 183GB each) * Model: DeepSeek-Math-V2 * Tensor Parallelism: 8 * SGLang Version: 0.5.8 #### 5.2.1 Latency Benchmark **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-Math-V2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.34 Total input tokens: 1972 Total input text tokens: 1972 Total generated tokens: 2784 Total generated tokens (retokenized): 2778 Request throughput (req/s): 0.19 Input token throughput (tok/s): 36.97 Output token throughput (tok/s): 52.19 Peak output token throughput (tok/s): 56.00 Peak concurrent requests: 3 Total token throughput (tok/s): 89.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5330.72 Median E2E Latency (ms): 5879.28 P90 E2E Latency (ms): 8320.33 P99 E2E Latency (ms): 9921.29 ---------------Time to First Token---------------- Mean TTFT (ms): 183.38 Median TTFT (ms): 177.92 P99 TTFT (ms): 217.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.96 Median TPOT (ms): 18.39 P99 TPOT (ms): 19.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.57 Median ITL (ms): 18.63 P95 ITL (ms): 19.26 P99 ITL (ms): 19.48 Max ITL (ms): 24.93 ================================================== ``` #### 5.2.2 Throughput Benchmark **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-Math-V2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 217.36 Total input tokens: 301701 Total input text tokens: 301701 Total generated tokens: 188375 Total generated tokens (retokenized): 187456 Request throughput (req/s): 4.60 Input token throughput (tok/s): 1388.05 Output token throughput (tok/s): 866.67 Peak output token throughput (tok/s): 2589.00 Peak concurrent requests: 109 Total token throughput (tok/s): 2254.72 Concurrency: 89.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19521.73 Median E2E Latency (ms): 12076.76 P90 E2E Latency (ms): 47248.87 P99 E2E Latency (ms): 86862.79 ---------------Time to First Token---------------- Mean TTFT (ms): 790.40 Median TTFT (ms): 456.81 P99 TTFT (ms): 4223.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 106.52 Median TPOT (ms): 107.24 P99 TPOT (ms): 238.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.29 Median ITL (ms): 38.34 P95 ITL (ms): 237.00 P99 ITL (ms): 347.49 Max ITL (ms): 3642.56 ================================================== ``` # DeepSeek-OCR Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR ## 1. Model Introduction [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks. **Key Features:** * **Advanced OCR**: High-accuracy text recognition from images and documents * **Multi-Modality**: Supports various image formats and document types **Available Models:** * **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks **License:** To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details. For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 OCR-Specific Prompts DeepSeek-OCR accepts recommended prompts from the model card: ```text theme={null} <|grounding|>Convert the document to markdown. ``` ```text theme={null} Free OCR. ``` **OpenAI-compatible image request example:** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "deepseek-ai/DeepSeek-OCR", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "\n<|grounding|>Convert the document to markdown." }, { "type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"} }, ], } ], "max_tokens": 512, } response = requests.post(url, json=data) print(response.text) ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (1x) * Model: DeepSeek-OCR * Tensor Parallelism: 1 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-OCR \ --tp 1 \ --dtype float16 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-OCR \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 4.45 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2770 Request throughput (req/s): 2.25 Input token throughput (tok/s): 442.89 Output token throughput (tok/s): 625.26 Peak output token throughput (tok/s): 635.00 Peak concurrent requests: 4 Total token throughput (tok/s): 1068.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 443.32 Median E2E Latency (ms): 493.29 ---------------Time to First Token---------------- Mean TTFT (ms): 21.59 Median TTFT (ms): 20.89 P99 TTFT (ms): 24.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.47 Median TPOT (ms): 1.52 P99 TPOT (ms): 1.53 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.52 Median ITL (ms): 1.51 P95 ITL (ms): 1.76 P99 ITL (ms): 1.93 Max ITL (ms): 8.28 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-OCR \ --tp 1 \ --ep 1 \ --dp 1 \ --enable-dp-attention \ --dtype float16 \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-OCR \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 16.24 Total input tokens: 301698 Total input text tokens: 301698 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 186927 Request throughput (req/s): 61.59 Input token throughput (tok/s): 18582.90 Output token throughput (tok/s): 11602.84 Peak output token throughput (tok/s): 15479.00 Peak concurrent requests: 179 Total token throughput (tok/s): 30185.75 Concurrency: 85.53 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1388.60 Median E2E Latency (ms): 901.43 ---------------Time to First Token---------------- Mean TTFT (ms): 73.36 Median TTFT (ms): 50.21 P99 TTFT (ms): 349.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.42 Median TPOT (ms): 7.31 P99 TPOT (ms): 27.99 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.04 Median ITL (ms): 4.62 P95 ITL (ms): 21.11 P99 ITL (ms): 36.92 Max ITL (ms): 172.15 ================================================== ``` # DeepSeek-OCR-2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2 ## 1. Model Introduction [DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats. **Key Features:** * **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents). * **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version. * **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks. * **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR. **Available Models:** * **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks **License:** To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details. For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs. **Note**: DeepSeek-OCR-2 has \~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details. ### 3.2 Configuration Tips * **Single GPU Deployment:** DeepSeek-OCR-2 (\~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving. * **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway). * **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage **OpenAI-compatible request example** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "deepseek-ai/DeepSeek-OCR-2", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "\n<|grounding|>Convert the document to markdown."}, {"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}}, ], } ], "max_tokens": 512, } response = requests.post(url, json=data) print(response.text) ``` **Reference** * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Recommended Prompts The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts). **Structured document conversion** — extracts text while preserving layout: ```text Example theme={null} <|grounding|>Convert the document to markdown. ``` **Free-form OCR** — extracts without layouts: ```text Example theme={null} Free OCR. ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (1x) * Model: DeepSeek-OCR-2 * Tensor Parallelism: 1 * sglang version: 0.0.0.dev1+g93fca0bbc We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models). #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-OCR-2 \ --enable-multimodal \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --model deepseek-ai/DeepSeek-OCR-2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 3.54 Total input tokens: 1972 Total input text tokens: 1972 Total generated tokens: 2784 Total generated tokens (retokenized): 2710 Request throughput (req/s): 2.83 Input token throughput (tok/s): 557.53 Output token throughput (tok/s): 787.10 Peak output token throughput (tok/s): 818.00 Peak concurrent requests: 5 Total token throughput (tok/s): 1344.63 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 352.69 Median E2E Latency (ms): 392.34 P90 E2E Latency (ms): 540.64 P99 E2E Latency (ms): 639.01 ---------------Time to First Token---------------- Mean TTFT (ms): 18.08 Median TTFT (ms): 16.57 P99 TTFT (ms): 25.67 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.18 Median TPOT (ms): 1.21 P99 TPOT (ms): 1.22 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.21 Median ITL (ms): 1.21 P95 ITL (ms): 1.28 P99 ITL (ms): 1.44 Max ITL (ms): 4.32 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-OCR-2 \ --enable-multimodal \ --tp 1 \ --ep 1 \ --dp 1 \ --enable-dp-attention \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --model deepseek-ai/DeepSeek-OCR-2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 14.79 Total input tokens: 301698 Total input text tokens: 301698 Total generated tokens: 188375 Total generated tokens (retokenized): 185236 Request throughput (req/s): 67.63 Input token throughput (tok/s): 20402.54 Output token throughput (tok/s): 12738.99 Peak output token throughput (tok/s): 17508.00 Peak concurrent requests: 187 Total token throughput (tok/s): 33141.53 Concurrency: 86.87 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1284.50 Median E2E Latency (ms): 866.07 P90 E2E Latency (ms): 3027.32 P99 E2E Latency (ms): 5490.63 ---------------Time to First Token---------------- Mean TTFT (ms): 86.08 Median TTFT (ms): 50.09 P99 TTFT (ms): 613.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.79 Median TPOT (ms): 6.54 P99 TPOT (ms): 50.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.42 Median ITL (ms): 4.64 P95 ITL (ms): 23.65 P99 ITL (ms): 39.62 Max ITL (ms): 452.65 ================================================== ``` # DeepSeek-R1 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-R1 ## 1. Model Introduction [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms. **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X * **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X * **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16) * **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8) * **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8) * **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ) * **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4) **License:** To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details. For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy. ### 3.2 Optimal Configurations Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware. ### 3.3 Configuration Tips DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply. **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code ``` **Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. **NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --reasoning-parser deepseek-r1 \ --tp 8 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \ --tp 8 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: 🔧 Tool Call: None Arguments: {"location": "Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-0528", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism. #### 4.2.4 Thinking Budget Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```shell Command theme={null} python3 -m sglang.launch_server \ --model deepseek-ai/DeepSeek-R1 \ --tp 8 \ --port 30000 \ --reasoning-parser deepseek-r1 \ --enable-custom-logit-processor ``` ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: B200 GPU (8x) * Model: DeepSeek-R1-0528 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off: * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 40.00 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4205 Request throughput (req/s): 0.25 Input token throughput (tok/s): 152.52 Output token throughput (tok/s): 105.24 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 257.76 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3998.40 Median E2E Latency (ms): 3207.53 ---------------Time to First Token---------------- Mean TTFT (ms): 153.00 Median TTFT (ms): 140.76 P99 TTFT (ms): 214.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.16 Median TPOT (ms): 9.15 P99 TPOT (ms): 9.21 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.16 Median ITL (ms): 9.15 P95 ITL (ms): 9.47 P99 ITL (ms): 9.63 Max ITL (ms): 15.45 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 51.21 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40458 Request throughput (req/s): 1.56 Input token throughput (tok/s): 774.66 Output token throughput (tok/s): 795.30 Peak output token throughput (tok/s): 1088.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1569.96 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8918.33 Median E2E Latency (ms): 9466.16 ---------------Time to First Token---------------- Mean TTFT (ms): 273.51 Median TTFT (ms): 131.71 P99 TTFT (ms): 839.57 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.56 Median TPOT (ms): 17.46 P99 TPOT (ms): 28.68 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.02 Median ITL (ms): 14.70 P95 ITL (ms): 16.41 P99 ITL (ms): 112.38 Max ITL (ms): 461.90 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 110.46 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251441 Request throughput (req/s): 4.53 Input token throughput (tok/s): 2261.80 Output token throughput (tok/s): 2282.90 Peak output token throughput (tok/s): 3900.00 Peak concurrent requests: 109 Total token throughput (tok/s): 4544.71 Concurrency: 92.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20380.71 Median E2E Latency (ms): 19391.65 ---------------Time to First Token---------------- Mean TTFT (ms): 563.14 Median TTFT (ms): 147.62 P99 TTFT (ms): 2632.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 40.11 Median TPOT (ms): 41.98 P99 TPOT (ms): 50.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 39.37 Median ITL (ms): 26.36 P95 ITL (ms): 98.16 P99 ITL (ms): 150.08 Max ITL (ms): 2052.85 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 411.34 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44390 Request throughput (req/s): 0.02 Input token throughput (tok/s): 14.83 Output token throughput (tok/s): 108.07 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 122.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 41132.04 Median E2E Latency (ms): 44288.71 ---------------Time to First Token---------------- Mean TTFT (ms): 125.76 Median TTFT (ms): 126.19 P99 TTFT (ms): 137.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.21 Median TPOT (ms): 9.20 P99 TPOT (ms): 9.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.23 Median ITL (ms): 9.22 P95 ITL (ms): 9.64 P99 ITL (ms): 9.86 Max ITL (ms): 15.18 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 348.93 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 317630 Request throughput (req/s): 0.23 Input token throughput (tok/s): 113.69 Output token throughput (tok/s): 912.02 Peak output token throughput (tok/s): 1088.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1025.70 Concurrency: 14.07 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 61360.70 Median E2E Latency (ms): 62071.20 ---------------Time to First Token---------------- Mean TTFT (ms): 176.02 Median TTFT (ms): 153.75 P99 TTFT (ms): 268.44 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.42 Median TPOT (ms): 15.59 P99 TPOT (ms): 16.07 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.39 Median ITL (ms): 15.17 P95 ITL (ms): 16.62 P99 ITL (ms): 18.13 Max ITL (ms): 226.59 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 589.31 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1297658 Request throughput (req/s): 0.54 Input token throughput (tok/s): 269.70 Output token throughput (tok/s): 2207.16 Peak output token throughput (tok/s): 2944.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2476.86 Concurrency: 57.03 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 105032.36 Median E2E Latency (ms): 108229.09 ---------------Time to First Token---------------- Mean TTFT (ms): 223.91 Median TTFT (ms): 158.15 P99 TTFT (ms): 474.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.94 Median TPOT (ms): 26.72 P99 TPOT (ms): 27.99 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.79 Median ITL (ms): 25.37 P95 ITL (ms): 26.70 P99 ITL (ms): 105.49 Max ITL (ms): 237.91 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 40.65 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4195 Request throughput (req/s): 0.25 Input token throughput (tok/s): 1031.65 Output token throughput (tok/s): 103.56 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1135.20 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4063.62 Median E2E Latency (ms): 3296.13 ---------------Time to First Token---------------- Mean TTFT (ms): 165.91 Median TTFT (ms): 154.96 P99 TTFT (ms): 240.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.26 Median TPOT (ms): 9.27 P99 TPOT (ms): 9.42 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.28 Median ITL (ms): 9.28 P95 ITL (ms): 9.66 P99 ITL (ms): 9.83 Max ITL (ms): 14.06 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 56.71 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41490 Request throughput (req/s): 1.41 Input token throughput (tok/s): 5290.75 Output token throughput (tok/s): 733.41 Peak output token throughput (tok/s): 1024.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6024.16 Concurrency: 14.25 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10098.99 Median E2E Latency (ms): 10623.46 ---------------Time to First Token---------------- Mean TTFT (ms): 486.80 Median TTFT (ms): 189.59 P99 TTFT (ms): 2138.73 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 19.06 Median TPOT (ms): 19.23 P99 TPOT (ms): 30.69 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.53 Median ITL (ms): 15.63 P95 ITL (ms): 16.64 P99 ITL (ms): 109.71 Max ITL (ms): 1471.36 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-R1-0528 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 115.55 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169275 Request throughput (req/s): 2.77 Input token throughput (tok/s): 11024.93 Output token throughput (tok/s): 1468.50 Peak output token throughput (tok/s): 2254.00 Peak concurrent requests: 70 Total token throughput (tok/s): 12493.43 Concurrency: 59.45 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21465.98 Median E2E Latency (ms): 20686.26 ---------------Time to First Token---------------- Mean TTFT (ms): 913.93 Median TTFT (ms): 224.92 P99 TTFT (ms): 6257.83 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 39.93 Median TPOT (ms): 40.99 P99 TPOT (ms): 60.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.83 Median ITL (ms): 26.29 P95 ITL (ms): 113.81 P99 ITL (ms): 176.94 Max ITL (ms): 5521.53 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py \ --num-shots 8 \ --num-questions 1316 \ --parallel 1316 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # DeepSeek-V3 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3 ## 1. Model Introduction [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost. Key highlights include: * **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable. * **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies. * **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively. * **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code ``` DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`. **Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/get-started/quickstart) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3 \ --reasoning-parser deepseek-v3 \ --tp 8 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To determine 15% of a number, follow these steps: **Step 1: Understand the Problem** You need to find 15% of a given number. Let's assume the number is 240 for this example. **Step 2: Convert the Percentage to a Decimal** To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100. \[ 15\% = \frac{15}{100} = 0.15 \] **Step 3: Multiply the Decimal by the Number** Now, multiply the decimal form of the percentage by the number you want to find the percentage of. \[ 0.15 \times 240 \] **Step 4: Perform the Multiplication** Calculate the product: \[ 0.15 \times 240 = 36 \] **Step 5: Conclusion** Therefore, 15% of 240 is: \boxed{36} The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3 \ --tool-call-parser deepseekv3 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Quick Test (curl):** ```shell Command theme={null} curl "http://127.0.0.1:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ -d '{ "temperature": 0, "max_tokens": 100, "model": "deepseek-ai/DeepSeek-V3", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}] }' ``` Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** Please attach the code blocks below to the previous Python script. ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8. **Enable with:** ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --speculative-algorithm EAGLE \ --trust-remote-code \ --tp 8 ``` The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48). The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 MLA Optimizations DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations: * **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency. * **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, CutlassMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default. * **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8. * **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency. * **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only). Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline. **Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details. #### 4.2.5 Multi-Node Deployment For multi-node serving and hardware-specific examples: * [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended) * [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3) * [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker) * [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes) * [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization) * [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization) * [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization) * [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus) * [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment) **Blog references for large-scale deployment:** * [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/)) * [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/) * [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/) ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: DeepSeek-V3 * Tensor Parallelism: 8 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --dp 8 \ --enable-dp-attention \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-V3 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 81.27 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2774 Request throughput (req/s): 0.12 Input token throughput (tok/s): 24.27 Output token throughput (tok/s): 34.26 Peak output token throughput (tok/s): 65.00 Peak concurrent requests: 2 Total token throughput (tok/s): 58.52 Concurrency: 1.00 Accept length: 2.61 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8123.17 Median E2E Latency (ms): 7982.65 ---------------Time to First Token---------------- Mean TTFT (ms): 1080.76 Median TTFT (ms): 1248.82 P99 TTFT (ms): 1896.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.04 Median TPOT (ms): 24.76 P99 TPOT (ms): 32.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.41 Median ITL (ms): 20.14 P95 ITL (ms): 60.28 P99 ITL (ms): 60.99 Max ITL (ms): 61.49 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --ep 8 \ --dp 8 \ --enable-dp-attention \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model deepseek-ai/DeepSeek-V3 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 406.16 Total input tokens: 301701 Total input text tokens: 301701 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 187542 Request throughput (req/s): 2.46 Input token throughput (tok/s): 742.81 Output token throughput (tok/s): 463.80 Peak output token throughput (tok/s): 1299.00 Peak concurrent requests: 109 Total token throughput (tok/s): 1206.61 Concurrency: 87.53 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 35552.98 Median E2E Latency (ms): 21466.07 ---------------Time to First Token---------------- Mean TTFT (ms): 1521.51 Median TTFT (ms): 476.80 P99 TTFT (ms): 8329.50 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 214.73 Median TPOT (ms): 152.00 P99 TPOT (ms): 1155.85 ---------------Inter-Token Latency---------------- Mean ITL (ms): 182.10 Median ITL (ms): 79.18 P95 ITL (ms): 398.60 P99 ITL (ms): 1488.96 Max ITL (ms): 43465.60 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000 ``` * **Test Results**: * DeepSeek-V3 ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 32.450 s Output throughput: 614.211 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} cd sglang bash benchmark/mmlu/download_data.sh python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 8000 ``` * **Test Results**: * DeepSeek-V3 ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.800 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.928 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.928 subject: college_biology, #q:144, acc: 0.965 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.800 subject: college_medicine, #q:173, acc: 0.861 Total latency: 58.339 Average accuracy: 0.871 ``` # DeepSeek-V3.1 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1 ## 1. Model Introduction [DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture. **[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply. **Recommended GPU configurations by weight type:**
Weight Type Supported Hardware
FP8 (recommended) 8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20
BF16 (upcast from FP8) 2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800
INT8 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3
W4A8 / AWQ / MXFP4 / NVFP4 8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200
> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it. **DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min): ```bash theme={null} python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code ``` DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`. **Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads. **NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/get-started/quickstart) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3.1-Terminus \ --reasoning-parser deepseek-v3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100. To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15. Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240. I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36. I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer. So, 15% of 240 is 36. The problem says "step by step," so I should present it clearly. =============== Content ================= To find 15% of 240, follow these steps: 1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \). 2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \). 3. Multiply the decimal by 240: \( 0.15 \times 240 \). 4. Perform the multiplication: - \( 0.15 \times 200 = 30 \) - \( 0.15 \times 40 = 6 \) - Add the results: \( 30 + 6 = 36 \). Alternatively, you can find 15% by breaking it into parts: - 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \). - 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \). - Add 10% and 5%: \( 24 + 12 = 36 \). Thus, 15% of 240 is 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model deepseek-ai/DeepSeek-V3.1-Terminus \ --tool-call-parser deepseekv31 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide. I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world. The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** Please attach the code blocks below to the previous Python script. ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.1-Terminus", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "Currently, it is **22°C and sunny** in Beijing." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: DeepSeek-V3.1-Terminus * Tensor Parallelism: 8 * sglang version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off: * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 106.24 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4201 Request throughput (req/s): 0.09 Input token throughput (tok/s): 57.43 Output token throughput (tok/s): 39.72 Peak output token throughput (tok/s): 43.00 Peak concurrent requests: 2 Total token throughput (tok/s): 97.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10620.29 Median E2E Latency (ms): 8868.09 ---------------Time to First Token---------------- Mean TTFT (ms): 557.85 Median TTFT (ms): 213.58 P99 TTFT (ms): 1625.28 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.84 Median TPOT (ms): 23.90 P99 TPOT (ms): 24.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.90 Median ITL (ms): 23.92 P95 ITL (ms): 24.15 P99 ITL (ms): 24.25 Max ITL (ms): 25.44 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 107.71 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40625 Request throughput (req/s): 0.74 Input token throughput (tok/s): 368.28 Output token throughput (tok/s): 378.84 Peak output token throughput (tok/s): 508.00 Peak concurrent requests: 19 Total token throughput (tok/s): 747.12 Concurrency: 13.72 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18473.65 Median E2E Latency (ms): 19558.42 ---------------Time to First Token---------------- Mean TTFT (ms): 607.91 Median TTFT (ms): 191.32 P99 TTFT (ms): 2135.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.50 Median TPOT (ms): 35.99 P99 TPOT (ms): 43.62 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.10 Median ITL (ms): 32.18 P95 ITL (ms): 33.03 P99 ITL (ms): 159.99 Max ITL (ms): 453.99 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 207.65 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 251238 Request throughput (req/s): 2.41 Input token throughput (tok/s): 1203.15 Output token throughput (tok/s): 1216.79 Peak output token throughput (tok/s): 2100.00 Peak concurrent requests: 106 Total token throughput (tok/s): 2419.94 Concurrency: 91.02 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 37800.20 Median E2E Latency (ms): 35921.56 ---------------Time to First Token---------------- Mean TTFT (ms): 835.15 Median TTFT (ms): 236.88 P99 TTFT (ms): 2868.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 73.33 Median TPOT (ms): 76.35 P99 TPOT (ms): 97.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 73.30 Median ITL (ms): 50.82 P95 ITL (ms): 180.67 P99 ITL (ms): 186.83 Max ITL (ms): 1661.39 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 1097.29 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44313 Request throughput (req/s): 0.01 Input token throughput (tok/s): 5.56 Output token throughput (tok/s): 40.52 Peak output token throughput (tok/s): 43.00 Peak concurrent requests: 2 Total token throughput (tok/s): 46.08 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 109725.52 Median E2E Latency (ms): 117748.67 ---------------Time to First Token---------------- Mean TTFT (ms): 156.67 Median TTFT (ms): 156.19 P99 TTFT (ms): 159.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.41 Median TPOT (ms): 24.51 P99 TPOT (ms): 24.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.65 Median ITL (ms): 24.58 P95 ITL (ms): 25.68 P99 ITL (ms): 25.93 Max ITL (ms): 29.80 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 775.02 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 317426 Request throughput (req/s): 0.10 Input token throughput (tok/s): 51.18 Output token throughput (tok/s): 410.70 Peak output token throughput (tok/s): 512.00 Peak concurrent requests: 18 Total token throughput (tok/s): 461.89 Concurrency: 13.86 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 134236.65 Median E2E Latency (ms): 135181.28 ---------------Time to First Token---------------- Mean TTFT (ms): 214.35 Median TTFT (ms): 194.12 P99 TTFT (ms): 300.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.72 Median TPOT (ms): 34.00 P99 TPOT (ms): 34.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.69 Median ITL (ms): 33.71 P95 ITL (ms): 34.50 P99 ITL (ms): 34.92 Max ITL (ms): 164.76 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1231.97 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1296845 Request throughput (req/s): 0.26 Input token throughput (tok/s): 129.01 Output token throughput (tok/s): 1056.05 Peak output token throughput (tok/s): 1472.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1185.07 Concurrency: 56.17 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 216256.25 Median E2E Latency (ms): 224192.84 ---------------Time to First Token---------------- Mean TTFT (ms): 317.68 Median TTFT (ms): 235.28 P99 TTFT (ms): 649.39 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.30 Median TPOT (ms): 55.10 P99 TPOT (ms): 56.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.13 Median ITL (ms): 52.95 P95 ITL (ms): 56.23 P99 ITL (ms): 181.04 Max ITL (ms): 208.61 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 114.47 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4194 Request throughput (req/s): 0.09 Input token throughput (tok/s): 366.39 Output token throughput (tok/s): 36.87 Peak output token throughput (tok/s): 42.00 Peak concurrent requests: 2 Total token throughput (tok/s): 403.26 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 11442.86 Median E2E Latency (ms): 9508.87 ---------------Time to First Token---------------- Mean TTFT (ms): 883.78 Median TTFT (ms): 481.38 P99 TTFT (ms): 2217.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.93 Median TPOT (ms): 25.05 P99 TPOT (ms): 26.11 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.08 Median ITL (ms): 25.08 P95 ITL (ms): 26.18 P99 ITL (ms): 26.28 Max ITL (ms): 27.41 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 162.33 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41443 Request throughput (req/s): 0.49 Input token throughput (tok/s): 1848.27 Output token throughput (tok/s): 256.70 Peak output token throughput (tok/s): 467.00 Peak concurrent requests: 19 Total token throughput (tok/s): 2104.97 Concurrency: 14.52 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 29456.89 Median E2E Latency (ms): 27628.16 ---------------Time to First Token---------------- Mean TTFT (ms): 1784.30 Median TTFT (ms): 1347.21 P99 TTFT (ms): 5384.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.65 Median TPOT (ms): 52.09 P99 TPOT (ms): 74.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.23 Median ITL (ms): 34.52 P95 ITL (ms): 35.81 P99 ITL (ms): 513.25 Max ITL (ms): 2865.73 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model deepseek-ai/DeepSeek-V3.1 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 282.55 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169081 Request throughput (req/s): 1.13 Input token throughput (tok/s): 4508.6 Output token throughput (tok/s): 601.67 Peak output token throughput (tok/s): 1216 Peak concurrent requests: 68 Total token throughput (tok/s): 5110.27 Concurrency: 59.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 52810.32 Median E2E Latency (ms): 50981.81 ---------------Time to First Token---------------- Mean TTFT (ms): 786.69 Median TTFT (ms): 499.38 P99 TTFT (ms): 2925.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 97.93 Median TPOT (ms): 103.45 P99 TPOT (ms): 157.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 98.11 Median ITL (ms): 55.7 P95 ITL (ms): 240.71 P99 ITL (ms): 1114.36 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py \ --num-shots 8 \ --num-questions 1316 \ --parallel 1316 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # DeepSeek-V3.2 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2 ## 1. Model Introduction The DeepSeek-V3.2 series includes three model variants, each optimized for different use cases: **[DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)** is an upgraded version of DeepSeek-V3.1-Terminus, introducing the DeepSeek Sparse Attention (DSA) mechanism through continued training. DSA is a fine-grained sparse attention mechanism powered by a lightning indexer, enabling DeepSeek-V3.2-Exp to achieve significant efficiency improvements in long-context scenarios. Recommended for general conversations, long-context processing, and efficient inference. **[DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)** is the standard version suitable for general tasks and conversational scenarios. For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for standard conversations and general tasks. **[DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale)** is a special variant designed exclusively for deep reasoning tasks. This model is specifically optimized for scenarios requiring complex logical reasoning and deep thinking. However this model does not support tool calls (see below). For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for deep reasoning tasks, complex logical problems, and mathematical reasoning. **[DeepSeek-V3.2-NVFP4](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4)** is an NVIDIA-optimized NVFP4-quantized variant of DeepSeek-V3.2 for Blackwell devices. It uses ModelOpt FP4 quantization with a choice of MoE runner backends (`flashinfer_trtllm` (recommended), `flashinfer_cutlass`, or `flashinfer_cutedsl`), enabling efficient deployment with lower tensor parallelism (TP=4). It supports the same features as DeepSeek-V3.2 including tool calling, reasoning, and speculative decoding (MTP). **[DeepSeek-V3.2-MXFP4](https://huggingface.co/amd/DeepSeek-V3.2-mxfp4)** is an OCP-MXFP4 optimized variant for DeepSeek-V3.2 for AMD MI300X/MI355X devices. It uses OCP MXFP4 quantization with a triton mxfp4 backend (the same backend for gptoss-120B), enabling efficient deployment with lower tensor parallelism (TP=8) in a single node. It includes the same features as DeepSeek-V3.2 including tool calling, reasoning, fp8-kv, CP, TP and speculative decoding MTP. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ### 2.1 Docker Images Pre-built Docker images are available for different hardware platforms: ```bash Command theme={null} # NVIDIA H200 / B200 docker pull lmsysorg/sglang:latest # AMD MI350 / MI355X docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x # AMD MI300X # Note: v0.5.8-rocm700-mi30x does not include PR #17504. # Prefer the newest MI30x ROCm image tag from Docker Hub when available, or build from source. docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x # Ascend NPU (Atlas 800I A2 / A3) docker pull lmsysorg/sglang:dsv32-a2 docker pull lmsysorg/sglang:dsv32-a3 ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. SGLang supports serving DeepSeek V3.2 on NVIDIA H200, B200, and AMD MI300X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on DeepSeek-V3.2. ### 3.2 Configuration Tips * **Short-sequence MHA prefill (adaptive):** For prefill sequences shorter than 2048 tokens (default threshold), the DSA backend automatically switches to standard MHA (using FlashAttention variable-length on SM90, TRT-LLM ragged MHA on SM100). To extend this to longer sequences set env var `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` to a larger value (potential minor accuracy trade-off). * **DSA prefill/decode attention kernels (`--dsa-prefill-backend`, `--dsa-decode-backend`):** The `dsa` backend is automatically selected for DeepSeek-V3.2. Available kernels: `flashmla_sparse`, `flashmla_sparse_q8` (native FP8 e4m3 sparse prefill — no fp8→bf16 dequantization round-trip; Hopper SM90 + `--kv-cache-dtype fp8_e4m3` only, prefill only), `flashmla_kv`, `flashmla_auto`, `fa3` (Hopper only), `tilelang` (GPU/HPU/NPU), `aiter` (AMD, decode only), `trtllm` (Blackwell only). Defaults: Hopper BF16 KV → `flashmla_sparse` prefill / `fa3` decode; Hopper FP8 KV → `flashmla_kv` both; Blackwell BF16 → `flashmla_sparse` / `trtllm`; Blackwell FP8 → `trtllm` both. * **Index Cache:** Reuses indexer results across layers for efficiency at negligible accuracy cost. For **GLM-5** specifically, append `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for a better speed/accuracy tradeoff. * **HiSparse (experimental):** Reduces per-request GPU memory during long-context decode by offloading KV data to CPU pinned memory. Requires PD disaggregation mode (decode instance only). See [HiSparse Guide](../../../docs/advanced_features/hisparse_guide). * **NVFP4 on Blackwell:** Specify `--quantization modelopt_fp4` and `--moe-runner-backend flashinfer_trtllm` (recommended) / `flashinfer_cutlass` / `flashinfer_cutedsl`. Full example: ```bash theme={null} python -m sglang.launch_server --model nvidia/DeepSeek-V3.2-NVFP4 --tp 4 \ --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 ``` * **NCCL timeout:** Slow model loading → add `--dist-timeout 3600`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [Basic API Usage](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser DeepSeek-V3.2 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --reasoning-parser deepseek-v3 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, extra_body = {"chat_template_kwargs": {"thinking": True}}, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they use different parameters. Enable the tool call parser: **Note:** DeepSeek-V3.2-Speciale does **NOT** support tool calling. Launch it with reasoning parser only: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Speciale \ --trust-remote-code \ --tp-size 8 --dp-size 8 --enable-dp-attention \ --reasoning-parser deepseek-v3 ``` **Deployment Command:** For DeepSeek-V3.2-Exp: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tool-call-parser deepseekv31 \ --reasoning-parser deepseek-v3 \ --chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` For DeepSeek-V3.2, use `--tool-call-parser deepseekv32` and remove `--chat-template`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, extra_body = {"chat_template_kwargs": {"thinking": True}}, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V3.2-Exp", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding) SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). This optimization significantly improves decoding speed for small batch sizes. **With DP Attention:** ```bash Command theme={null} python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 \ --enable-dp-attention \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` **With Pure TP:** ```bash Command theme={null} python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` Find optimal values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. `--max-running-requests` defaults to 48 for MTP. Increase it for larger batch sizes. The spec-v2 overlap scheduler is enabled by default. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 PD Disaggregation Prefill-Decode (PD) disaggregation separates prefill and decode stages onto different instances, improving GPU utilization for mixed workloads. **Prefill command:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --disaggregation-mode prefill \ --host $LOCAL_IP \ --port $PORT \ --tp 8 \ --dp 8 \ --enable-dp-attention \ --dist-init-addr ${HOST}:${DIST_PORT} \ --trust-remote-code \ --disaggregation-bootstrap-port 8998 \ --mem-fraction-static 0.9 ``` **Decode command:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --disaggregation-mode decode \ --host $LOCAL_IP \ --port $PORT \ --tp 8 \ --dp 8 \ --enable-dp-attention \ --dist-init-addr ${HOST}:${DIST_PORT} \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Router command:** ```bash Command theme={null} python -m sglang_router.launch_router --pd-disaggregation \ --prefill $PREFILL_ADDR 8998 \ --decode $DECODE_ADDR \ --host 127.0.0.1 \ --port 30000 ``` For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi\_node\_deployment docs](../../../docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd). #### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP SGLang provides two context parallel (CP) modes for long-sequence workloads, controlled with `--dsa-prefill-cp-mode`. **In-sequence splitting** (`--dsa-prefill-cp-mode in-seq-split`): Each CP rank handles a uniform shard of the sequence; KV cache is gathered via all-gather. Batch size is restricted to 1 during prefill. See [PR #12065](https://github.com/sgl-project/sglang/pull/12065). ```bash Command theme={null} # In-seq splitting mode — EP + DP, batch size 1 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 --ep 8 --dp 2 --enable-dp-attention \ --enable-dsa-prefill-context-parallel --attn-cp-size 4 \ --dsa-prefill-cp-mode in-seq-split --max-running-requests 32 ``` **Round-robin splitting** (`--dsa-prefill-cp-mode round-robin-split`, default): Distributes tokens by `token_idx % cp_size`. Supports fused MoE, FP8 KV cache, and multi-batch prefill. Cannot be combined with DP attention. See [PR #13959](https://github.com/sgl-project/sglang/pull/13959). ```bash Command theme={null} # Round-robin splitting — FusedMoE + CP8 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 --enable-dsa-prefill-context-parallel --attn-cp-size 8 \ --dsa-prefill-cp-mode round-robin-split --max-running-requests 32 ``` **PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper: We suggested `DP2` + `MTP` for local deployment of agentic workflow with DeepSeek V3.2 on Hopper platform: ```shell Command theme={null} export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32 export SGLANG_SET_CPU_AFFINITY=1 # Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970 # dp 2 : 5019.54 toks/sec, MAX ITL 7233 # dp 4 : 4942.82 toks/sec, MAX ITL 35654 # dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \ --dp 2 --enable-dp-attention --page-size 64 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-dp2-mtp ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` **CP + PP + EP + DP** `CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment: ```shell Command theme={null} # verified on Hopper platform sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --dp 1 --enable-dp-attention \ --moe-a2a-backend deepep --ep-size 16 \ --page-size 128 \ --chunked-prefill-size 16384 \ --attention-backend dsa \ --dsa-prefill-backend flashmla_sparse \ --dsa-decode-backend flashmla_sparse \ --enable-dsa-prefill-context-parallel \ --dsa-prefill-cp-mode round-robin-split \ --cuda-graph-max-bs-decode 128 \ --max-running-requests 128 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-dsa-pp-cp-ep-dp ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` **fp8 KV + CP + PP** With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes: ```shell Command theme={null} # verified in Hopper platform dp=1 dp_config=" \ --dp 1 --enable-dp-attention \ " cp_config=" \ --enable-dsa-prefill-context-parallel \ " if [ "$dp" -eq 1 ]; then cp_config=" \ $cp_config \ --dsa-prefill-cp-mode round-robin-split \ " else cp_config=" \ $cp_config \ --dsa-prefill-cp-mode in-seq-split \ " fi # see discussion : https://github.com/sgl-project/sglang/pull/12065 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \ $dp_config \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ $cp_config \ --kv-cache-dtype fp8_e4m3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-fp8kv-pp2-cp4 ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` ## 5. Benchmark ### 5.1 Speed Benchmark on Blackwell **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: DeepSeek-V3.2-Exp * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2-Exp \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 29.11 Total input tokens: 1972 Total input text tokens: 1972 Total input vision tokens: 0 Total generated tokens: 2784 Total generated tokens (retokenized): 2777 Request throughput (req/s): 0.34 Input token throughput (tok/s): 67.73 Output token throughput (tok/s): 95.62 Peak output token throughput (tok/s): 157.00 Peak concurrent requests: 3 Total token throughput (tok/s): 163.36 Concurrency: 1.00 Accept length: 2.46 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2909.74 Median E2E Latency (ms): 3088.27 P90 E2E Latency (ms): 4200.62 P99 E2E Latency (ms): 5588.52 ---------------Time to First Token---------------- Mean TTFT (ms): 317.58 Median TTFT (ms): 191.31 P99 TTFT (ms): 740.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.09 Median TPOT (ms): 9.25 P99 TPOT (ms): 11.73 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.35 Median ITL (ms): 7.64 P95 ITL (ms): 22.81 P99 ITL (ms): 23.33 Max ITL (ms): 31.45 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.2-Exp \ --tp 8 \ --ep 8 \ --dp 8 \ --enable-dp-attention \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2-Exp \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 219.09 Total input tokens: 301701 Total input text tokens: 301701 Total input vision tokens: 0 Total generated tokens: 188375 Total generated tokens (retokenized): 187443 Request throughput (req/s): 4.56 Input token throughput (tok/s): 1377.06 Output token throughput (tok/s): 859.80 Peak output token throughput (tok/s): 2465.00 Peak concurrent requests: 109 Total token throughput (tok/s): 2236.86 Concurrency: 88.05 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19291.23 Median E2E Latency (ms): 11927.39 ---------------Time to First Token---------------- Mean TTFT (ms): 530.36 Median TTFT (ms): 444.00 P99 TTFT (ms): 1504.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 106.16 Median TPOT (ms): 106.69 P99 TPOT (ms): 221.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.46 Median ITL (ms): 41.73 P95 ITL (ms): 225.67 P99 ITL (ms): 392.37 Max ITL (ms): 975.03 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 30000 ``` * **Test Results**: * DeepSeek-V3.2-Exp ``` Accuracy: 0.980 Invalid: 0.000 Latency: 19.128 s Output throughput: 965.919 token/s ``` * **Full GSM8K (1319 questions)** — for a stricter accuracy check, run the full set 8-shot: ```shell Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319 ``` * 8-shot: ``` Accuracy: 0.956 Invalid: 0.000 Latency: 25.109 s Output throughput: 5226.235 token/s ``` * 20-shot (long-context; stays close to the 8-shot result): ``` Accuracy: 0.956 Invalid: 0.000 Latency: 29.545 s Output throughput: 4418.617 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} cd sglang bash benchmark/mmlu/download_data.sh python3 benchmark/mmlu/bench_sglang.py --nsub 10 --port 30000 ``` * **Test Results**: * DeepSeek-V3.2-Exp ``` subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.961 subject: business_ethics, #q:100, acc: 0.860 subject: clinical_knowledge, #q:265, acc: 0.925 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.660 subject: college_computer_science, #q:100, acc: 0.880 subject: college_mathematics, #q:100, acc: 0.840 subject: college_medicine, #q:173, acc: 0.879 Total latency: 7.961 Average accuracy: 0.879 ``` #### 5.2.3 GPQA-Diamond Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --thinking-mode deepseek-v3 ``` * **Test Results** (model: `deepseek-ai/DeepSeek-V3.2-Exp`, 8×B200): * Default (`temperature=0`): mean **0.797** over 8 runs — closely matches the official GPQA-Diamond score of **79.9** for DeepSeek-V3.2-Exp reported in its [model card](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp) * With `temperature=1.0, top_p=0.95` (as recommended by DeepSeek): ```shell Command theme={null} python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 --max-tokens 128000 --repeat 8 --top-p 0.95 --temperature 1.0 --thinking-mode deepseek-v3 ``` ``` Repeat: 8, mean: 0.840 Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848'] ``` #### 5.2.4 AIME 2025 Benchmark Results on AIME 2025 (8×B200), evaluated with [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills.git):
Model pass\@1 avg-of-4 majority\@4 pass\@4
DeepSeek-V3.2-Exp 87.50% ± 1.67% 90.00% 90.00%
DeepSeek-V3.2 92.50% ± 1.67% 94.71% 96.67%
DeepSeek-V3.2-Speciale 95.00% ± 1.92% 95.83% 100.00%
**Reproduction.** Install [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills), launch the server with the tool-call and reasoning parsers, then run `ns eval`: ```bash Command theme={null} pip install git+https://github.com/NVIDIA/NeMo-Skills.git --ignore-installed blinker export NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECK=1 ns prepare_data aime25 ns eval \ --benchmarks=aime25:4 \ --server_type=sglang \ --model=deepseek-ai/DeepSeek-V3.2-Exp \ --server_address=http://localhost:30000/v1 \ --output_dir=nemo_skills_aime25_output \ ++chat_template_kwargs.thinking=true \ ++inference.temperature=1.0 \ ++inference.top_p=0.95 \ ++inference.tokens_to_generate=64000 # Use ++inference.tokens_to_generate=120000 for the DeepSeek-V3.2-Speciale model ``` ### 5.3 Speed Benchmark on Hopper **Test Environment:** * Hardware: NVIDIA H800 GPU (16x) * Model: DeepSeek-V3.2 * Tensor Parallelism: 16 * sglang version: 0.5.9 #### 5.3.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32 export SGLANG_SET_CPU_AFFINITY=1 # Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970 # dp 2 : 5019.54 toks/sec, MAX ITL 7233 # dp 4 : 4942.82 toks/sec, MAX ITL 35654 # dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \ --dp 2 --enable-dp-attention --page-size 64 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ --log-requests \ --context-length 65536 --max-running-requests 128 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ --allow-auto-truncate --enable-metrics \ --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \ --served-model-name DeepSeek-V3.2-Opt-dp2-mtp ) sglang_args=($sglang_args) sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host $MASTER_ADDR \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 48.96 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4217 Request throughput (req/s): 0.20 Input token throughput (tok/s): 124.62 Output token throughput (tok/s): 86.20 Peak output token throughput (tok/s): 113.00 Peak concurrent requests: 2 Total token throughput (tok/s): 210.81 Concurrency: 1.00 Accept length: 3.27 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4893.12 Median E2E Latency (ms): 3742.47 P90 E2E Latency (ms): 8877.37 P99 E2E Latency (ms): 10769.85 ---------------Time to First Token---------------- Mean TTFT (ms): 199.88 Median TTFT (ms): 176.15 P99 TTFT (ms): 272.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.99 Median TPOT (ms): 10.88 P99 TPOT (ms): 13.93 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.15 Median ITL (ms): 8.86 P95 ITL (ms): 17.29 P99 ITL (ms): 33.71 Max ITL (ms): 36.84 ================================================== ``` #### 5.3.2 Throughput-Sensitive Benchmark We simply use the same deployment method and vary the throughput by maximizing concurrencies: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host $MASTER_ADDR \ --port 30000 \ --model deepseek-ai/DeepSeek-V3.2 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 2048 \ --max-concurrency 1024 # see picture below why we use 1024 for concurrency, hence num prompts 2048 ``` DeepSeek 3.2 can steadily support concurrency up to `1024` and when concurrency is greater than `128`, the TTFT increase sharply: ![DeepSeek V3.2 Concurrency ISL/OSL=1024/128](https://github.com/user-attachments/assets/d5c9c9fb-44f3-4793-a0fd-f8fa954546f5) Performance record: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1024 Successful requests: 2048 Benchmark duration (s): 408.09 Total input tokens: 1048992 Total input text tokens: 1048992 Total generated tokens: 1032734 Total generated tokens (retokenized): 1031817 Request throughput (req/s): 5.02 Input token throughput (tok/s): 2570.50 Output token throughput (tok/s): 2530.66 Peak output token throughput (tok/s): 5092.00 Peak concurrent requests: 1035 Total token throughput (tok/s): 5101.16 Concurrency: 763.41 Accept length: 3.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 152117.70 Median E2E Latency (ms): 181704.84 P90 E2E Latency (ms): 215924.77 P99 E2E Latency (ms): 231679.59 ---------------Time to First Token---------------- Mean TTFT (ms): 127729.28 Median TTFT (ms): 170098.94 P99 TTFT (ms): 185705.73 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 49.18 Median TPOT (ms): 48.48 P99 TPOT (ms): 77.24 ---------------Inter-Token Latency---------------- Mean ITL (ms): 48.46 Median ITL (ms): 52.11 P95 ITL (ms): 110.26 P99 ITL (ms): 200.63 Max ITL (ms): 2666.37 ================================================== ``` By adding `--random-range-ratio 1`, we could get even higher statistical numbers: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: 64.0 Max request concurrency: 1024 Successful requests: 2048 Benchmark duration (s): 612.87 Total input tokens: 2097152 Total input text tokens: 2097152 Total generated tokens: 2097152 Total generated tokens (retokenized): 2096201 Request throughput (req/s): 3.34 Input token throughput (tok/s): 3421.84 Output token throughput (tok/s): 3421.84 Peak output token throughput (tok/s): 9077.00 Peak concurrent requests: 1039 Total token throughput (tok/s): 6843.68 Concurrency: 772.66 Accept length: 3.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 231222.27 Median E2E Latency (ms): 289846.24 P90 E2E Latency (ms): 314480.41 P99 E2E Latency (ms): 320392.27 ---------------Time to First Token---------------- Mean TTFT (ms): 194081.02 Median TTFT (ms): 252945.22 P99 TTFT (ms): 279637.50 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 36.31 Median TPOT (ms): 36.73 P99 TPOT (ms): 46.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 36.31 Median ITL (ms): 23.18 P95 ITL (ms): 96.79 P99 ITL (ms): 135.81 Max ITL (ms): 3121.00 ================================================== ``` # DeepSeek-V4 Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Pro, and Pro Official (0813). ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces): **NVIDIA GPUs** A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). ```bash Command theme={null} docker pull lmsysorg/sglang:latest docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ sglang serve ``` **AMD GPUs (ROCm)** AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. For example: * **MI355X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260822` * **MI300X** → `lmsysorg/sglang-rocm:v0.5.17-rocm720-mi30x-20260822` ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 docker run \ --device=/dev/kfd --device=/dev/dri \ --group-add video \ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ --shm-size 32g --ipc=host \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ lmsysorg/sglang-rocm:v0.5.17-rocm720-{mi35x,mi30x}-20260822 \ sglang serve ``` Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN, NODE\_RANK, NODE0\_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: * **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode). * **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below). Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction **DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head:
Variant Total params Active (MoE) Use
DeepSeek-V4-Flash 284B 13B single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)
DeepSeek-V4-Flash-0731 304 13B Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200
DeepSeek-V4-Pro 1.6T 49B high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)
DeepSeek-V4-Pro-0813 1.65T 49B Pro Official (0813), with a bundled DSpark draft head; verified on 4×GB300 (TP=4) · B200 / B300 / H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H100 (2-node, TP=16) · MI355X
The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling. **Highlights:** hybrid CSA + HCA attention (\~27% inference FLOPs / \~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar. **Recommended generation:** `temperature=1.0`, `top_p=1.0`. **Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). ## 2. Configuration Tips **Concurrency & DeepEP dispatch buffer** Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together. The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised. **Speculative decoding** The original Flash and Pro recipes use EAGLE. Flash Official (0731) and Pro Official (0813) use the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes. Do not use EAGLE on the checkpoints that bundle a DSpark head. On 0813, `--speculative-algorithm EAGLE` starts and serves without any error, but the draft head it binds accepts nothing — every decode batch logs `accept len: 1.00, accept rate: 0.00`, so you pay the draft cost for zero speedup. Output stays correct, which is what makes it easy to miss. Switch to `--speculative-algorithm DSPARK`; the startup log then reports `Draft checkpoint bundles a DSpark head`. For the original Flash and Pro checkpoints: * `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. * `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch. * `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves. * MTP runs on the v2 speculative path. **Compressed attention state dtype** DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path: ```bash Command theme={null} SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \ sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash \ ``` This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior. **EPLB + Waterfill (Experimental)** For recorded/static EPLB reproduction, first record an expert-distribution file by following [Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models). For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as the initial expert location. **Please checkout to latest main branch for this feature.** For non-PD reproduction, use: ```bash Command theme={null} --moe-a2a-backend deepep \ --deepep-mode auto \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill ``` For PD-Disagg reproduction, use `normal` mode on the prefill server and `low_latency` mode on the decode server. Add the same `--init-expert-location` flag to both commands: ```bash Command theme={null} # prefill --moe-a2a-backend deepep \ --deepep-mode normal \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill # decode --moe-a2a-backend deepep \ --deepep-mode low_latency \ --init-expert-location /path/to/expert_distribution_recorder_*.pt \ --enable-waterfill ``` You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize EPLB placement. Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe --enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the fused shared expert slot. **FP4 Indexer (Experimental)** DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial. ```bash Command theme={null} # Please use the latest main branch for this feature. sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash \ --tp 4 \ --moe-runner-backend flashinfer_mxfp4 \ --enable-deepseek-v4-fp4-indexer ``` **NVFP4 Hybrid Checkpoints** The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and [`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints quantize MoE experts to **NVFP4** while keeping attention and dense layers in **FP8**. It requires `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided. ```bash Command theme={null} sglang serve \ --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \ --tp 8 ``` or ```bash Command theme={null} sglang serve \ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \ --tp 8 ``` Requires Blackwell (SM100+). The MTP layer in this checkpoint stays MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path automatically. **Hopper (H100 / H200) note** Two options are available for running DeepSeek-V4 on Hopper: * **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16). * **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes). On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below. PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1` (or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake can discover the IB HCAs; without IB exposure mooncake silently falls back to TCP, which can lead to garbled KV transfer on large checkpoints. **RTX PRO 6000 (SM120 / Blackwell Desktop) note** RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner. V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes. HiCache and MegaMoE are **not** supported on RTX PRO 6000. **AMD (MI300X / MI355X) note** * **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`. * **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node. * **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`. * **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. * **Kernels** — uses the Unified KV attention and the flydsl MoE. **MegaMoE** MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput on MoE layers. To enable it, use the **MegaMoE** chip in the Playground below — the playground will swap `--moe-a2a-backend deepep` for `--moe-a2a-backend megamoe` and add the relevant launch settings automatically. Two variants are exposed: * **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations). * **W4A4** — adds `--enable-w4a4-mxfp4-megamoe` to run the custom W4A4 kernel (FP4 activations). The flag configures the required DeepGEMM settings. Higher throughput with negligible accuracy drop (\~89.5 GPQA on Pro). Notes: * The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead. * MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it. * When running MegaMoE, don't set `--moe-runner-backend` manually. * Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput). **SM90 (Hopper) FP8 MegaMoE (Experimental)** On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm` build with SM90 FP8 MegaMoE support. **Please use the latest image for this feature.** Enable the MegaMoE path with `--moe-a2a-backend megamoe` ```bash Command theme={null} SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \ SGLANG_DSV4_FP4_EXPERTS=0 \ sglang serve \ --model-path sgl-project/DeepSeek-V4-Flash-FP8 \ --tp 8 \ --moe-a2a-backend megamoe \ --chunked-prefill-size 4096 ``` `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used for batches at or below this cap. The right value depends on your parallelism / token-split scheme, and larger values reserve more HBM. **GB300 PD-Disagg cross-pod MNNVL** On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` to both prefill and decode `sglang serve` commands. ## 3. Advanced Usage ### 3.1 Reasoning Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, extra_body={"chat_template_kwargs": {"thinking": True}}, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` ```text Output theme={null} We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.To find 15% of 240, follow these steps: **Step 1: Understand the meaning of percent** "Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \). **Step2: Convert the percentage to a decimal or fraction** \( 15\% = \frac{15}{100} = 0.15 \) **Step3: Multiply by the given number** Multiply the decimal form by 240: \( 0.15 \times 240 \) **Step4: Perform the multiplication** \( 0.15 \times 240 = 36 \) **Answer:** 15% of 240 is **36**. ``` ### 3.2 Tool Calling Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, extra_body={"chat_template_kwargs": {"thinking": True}}, stream=True, ) thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if getattr(delta, "tool_calls", None): if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {"name": None, "arguments": ""} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]["name"] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` ```text Output theme={null} The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default. <｜DSML｜tool_calls> <｜DSML｜invoke name="get_weather"> <｜DSML｜parameter name="location" string="true">Beijing ``` ### 3.3 HiCache (Hierarchical KV Caching) HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: * **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. * **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices). For AMD devices, * **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout. * **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices). The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow. For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). ### 3.4 DSpark (Speculative Decoding) Flash Official (0731) and Pro Official (0813) bundle a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731` and `deepseek-ai/DeepSeek-V4-Pro-0813`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`. Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint. The Pro Official (0813) low-latency speed numbers in the Deploy panel were measured with `SGLANG_SIMULATE_ACC_LEN=4`, which pins the DSpark accept length at exactly 4.00. The recipe as shipped earns **4.678** on the same engine, so those rows read slightly conservative. The GSM8K figure for that cell is from the shipped command. The verified 4×GB300 FP4 low-latency command is: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \ --tp 4 \ --moe-runner-backend flashinfer_mxfp4 \ --speculative-algorithm DSPARK \ --mem-fraction-static 0.90 \ --chunked-prefill-size 4096 \ --swa-full-tokens-ratio 0.1 \ --host 0.0.0.0 \ --port 30000 ``` Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16. **Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. Both the 0731 and 0813 checkpoints resolve to five proposed tokens (the startup log reports `gamma=5, verify_num_draft_tokens=6`), which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five. Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone. For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override. DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. # Ernie4.5 Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5 ## 1. Model Introduction The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios. ERNIE-4.5 delivers advanced features as below: * **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks. * **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing. * **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations. * **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions. * **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, and MI355X GPUs. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ## 4. API Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT. ```shell Command theme={null} python -m sglang.launch_server \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --tp 1 ``` **Basic Python Client Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="baidu/ERNIE-4.5-21B-A3B-PT", messages=[ {"role": "user", "content": "What is artificial intelligence?"} ], temperature=1.0, top_p=0.95, max_tokens=1024 ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} **Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence. ### Key Characteristics of AI: ... ### In Summary: AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility. ``` **Streaming Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="baidu/ERNIE-4.5-21B-A3B-PT", messages=[ {"role": "user", "content": "Explain quantum computing in simple terms."} ], temperature=1.0, top_p=0.95, max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} Sure! Here’s a simple explanation of quantum computing: ### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules** 1. **Classic vs. Quantum Computers** - **Normal computers** use **bits** (0s and 1s) to store and process information. - **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**). 2. **Superposition: The Magic Behind Speed** - A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air. - Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems. 3. **Entanglement: Making Qubits Link** - When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein). - Entanglement allows quantum computers to process information in **very efficient ways**. 4. **What Quantum Computers Are Good At** - **Cracking encryption** (like RSA). - **Factoring large numbers** (used in encryption and cryptography). - **Searching unsorted databases** (way faster than classical computers). - **Simulating quantum systems** (like molecules for drug discovery). - **Optimizing problems** (like logistics or finance). 5. **Challenges & Current State** - Qubits are **fragile** and easily disturbed (called **decoherence**). - Engineers are working to keep qubits stable long enough to do useful calculations. - Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers. ### **Final Thought** Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity. Would you like an example of how a quantum computer might solve a problem? 😊 ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (1x) * Model: ERNIE-4.5-21B-A3B-PT * Tensor Parallelism: 1 * SGLang Version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```bash Command theme={null} python -m sglang.launch_server \ --model-path baidu/ERNIE-4.5-21B-A3B-PT \ --tp 1 ``` ##### 5.1.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 58.72 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4219 Request throughput (req/s): 0.17 Input token throughput (tok/s): 103.90 Output token throughput (tok/s): 71.87 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 175.77 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5869.86 Median E2E Latency (ms): 1870.80 ---------------Time to First Token---------------- Mean TTFT (ms): 4152.58 Median TTFT (ms): 36.81 P99 TTFT (ms): 37498.23 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.07 Median TPOT (ms): 4.09 P99 TPOT (ms): 4.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.08 Median ITL (ms): 4.08 P95 ITL (ms): 4.14 P99 ITL (ms): 4.20 Max ITL (ms): 4.67 ================================================== ``` ##### 5.1.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 34.30 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40773 Request throughput (req/s): 2.33 Input token throughput (tok/s): 1156.62 Output token throughput (tok/s): 1189.77 Peak output token throughput (tok/s): 1392.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2346.39 Concurrency: 14.14 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6060.62 Median E2E Latency (ms): 6496.70 ---------------Time to First Token---------------- Mean TTFT (ms): 78.90 Median TTFT (ms): 45.90 P99 TTFT (ms): 234.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.99 Median TPOT (ms): 12.16 P99 TPOT (ms): 14.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.75 Median ITL (ms): 11.48 P95 ITL (ms): 12.24 P99 ITL (ms): 34.85 Max ITL (ms): 105.01 ================================================== ``` ##### 5.1.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 66.63 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252449 Request throughput (req/s): 7.50 Input token throughput (tok/s): 3749.79 Output token throughput (tok/s): 3792.28 Peak output token throughput (tok/s): 4902.00 Peak concurrent requests: 113 Total token throughput (tok/s): 7542.06 Concurrency: 90.33 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12036.90 Median E2E Latency (ms): 11782.16 ---------------Time to First Token---------------- Mean TTFT (ms): 104.86 Median TTFT (ms): 84.62 P99 TTFT (ms): 297.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.89 Median TPOT (ms): 24.62 P99 TPOT (ms): 26.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.66 Median ITL (ms): 20.48 P95 ITL (ms): 45.57 P99 ITL (ms): 54.31 Max ITL (ms): 185.12 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 185.11 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44423 Request throughput (req/s): 0.05 Input token throughput (tok/s): 32.96 Output token throughput (tok/s): 240.19 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 273.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18508.84 Median E2E Latency (ms): 19866.81 ---------------Time to First Token---------------- Mean TTFT (ms): 32.59 Median TTFT (ms): 32.14 P99 TTFT (ms): 38.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.13 Median TPOT (ms): 4.13 P99 TPOT (ms): 4.20 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.16 Median ITL (ms): 4.12 P95 ITL (ms): 4.31 P99 ITL (ms): 4.36 Max ITL (ms): 7.28 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 263.48 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 317984 Request throughput (req/s): 0.30 Input token throughput (tok/s): 150.55 Output token throughput (tok/s): 1208.09 Peak output token throughput (tok/s): 1408.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1358.64 Concurrency: 14.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 47249.55 Median E2E Latency (ms): 47828.67 ---------------Time to First Token---------------- Mean TTFT (ms): 62.77 Median TTFT (ms): 57.10 P99 TTFT (ms): 93.70 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.92 Median TPOT (ms): 12.09 P99 TPOT (ms): 12.50 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.86 Median ITL (ms): 12.04 P95 ITL (ms): 12.68 P99 ITL (ms): 13.61 Max ITL (ms): 39.94 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 428.30 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1299877 Request throughput (req/s): 0.75 Input token throughput (tok/s): 371.09 Output token throughput (tok/s): 3037.63 Peak output token throughput (tok/s): 3880.00 Peak concurrent requests: 69 Total token throughput (tok/s): 3408.73 Concurrency: 57.08 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 76392.58 Median E2E Latency (ms): 79698.73 ---------------Time to First Token---------------- Mean TTFT (ms): 92.79 Median TTFT (ms): 78.71 P99 TTFT (ms): 168.89 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.81 Median TPOT (ms): 19.15 P99 TPOT (ms): 19.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.77 Median ITL (ms): 18.77 P95 ITL (ms): 19.86 P99 ITL (ms): 42.08 Max ITL (ms): 74.36 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.59 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4216 Request throughput (req/s): 0.54 Input token throughput (tok/s): 2256.43 Output token throughput (tok/s): 227.04 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 2 Total token throughput (tok/s): 2483.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1856.72 Median E2E Latency (ms): 1513.87 ---------------Time to First Token---------------- Mean TTFT (ms): 86.66 Median TTFT (ms): 72.30 P99 TTFT (ms): 167.13 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.19 Median TPOT (ms): 4.22 P99 TPOT (ms): 4.30 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.20 Median ITL (ms): 4.23 P95 ITL (ms): 4.34 P99 ITL (ms): 4.42 Max ITL (ms): 5.68 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 40.25 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41646 Request throughput (req/s): 1.99 Input token throughput (tok/s): 7454.72 Output token throughput (tok/s): 1035.37 Peak output token throughput (tok/s): 1310.00 Peak concurrent requests: 20 Total token throughput (tok/s): 8490.09 Concurrency: 14.37 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7229.56 Median E2E Latency (ms): 7578.95 ---------------Time to First Token---------------- Mean TTFT (ms): 137.38 Median TTFT (ms): 122.59 P99 TTFT (ms): 485.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.04 Median TPOT (ms): 14.24 P99 TPOT (ms): 20.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.64 Median ITL (ms): 12.36 P95 ITL (ms): 14.72 P99 ITL (ms): 57.39 Max ITL (ms): 411.31 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model baidu/ERNIE-4.5-21B-A3B-PT \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 78.33 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169888 Request throughput (req/s): 4.09 Input token throughput (tok/s): 16262.33 Output token throughput (tok/s): 2170.20 Peak output token throughput (tok/s): 3005.00 Peak concurrent requests: 73 Total token throughput (tok/s): 18432.53 Concurrency: 58.79 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14392.52 Median E2E Latency (ms): 14460.70 ---------------Time to First Token---------------- Mean TTFT (ms): 184.82 Median TTFT (ms): 155.24 P99 TTFT (ms): 379.82 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.97 Median TPOT (ms): 28.31 P99 TPOT (ms): 33.61 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.79 Median ITL (ms): 20.55 P95 ITL (ms): 47.55 P99 ITL (ms): 145.64 Max ITL (ms): 287.62 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py \ --num-shots 8 \ --num-questions 1316 \ --parallel 1316 ``` * Test Results: * ERNIE-4.5-21B-A3B-PT ``` Accuracy: 0.865 Invalid: 0.000 Latency: 21.669 s Output throughput: 10359.790 token/s ``` # Ernie4.5-VL Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5-VL ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-ernie4-5-vl-guide # Edit this file and submit a PR ``` ## 📚 Reference * [GLM-4.6V](../GLM/GLM-4.6V) *** **Let's build this together!** 🌟 # Chroma-1.0 Source: https://docs.sglang.io/cookbook/autoregressive/FlashLabs/Chroma1.0 ## 1. Model Introduction [Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities: * **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios. * **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics. * **End-to-End Architecture**: Provides a complete processing workflow from speech to speech. * **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content. ## 2. Architecture Overview **Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by: 1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops. 2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations. 3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable. Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported. * **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic) * **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable) ## 3. Installation & Setup We recommend following these steps to set up the environment and prepare the model. ### Step 1: Get the Docker Image Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured. ```bash Command theme={null} docker pull flashlabs/chroma:latest ``` ### Step 2: Download Model Weights Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods: **Method 1: Using Python (Recommended)** ```bash Command theme={null} huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B ``` **Method 2: Using Git Clone** Make sure you have Git LFS installed before cloning. ```bash Command theme={null} # Install Git LFS first git lfs install # Clone the repository git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B ``` ### Step 3: Download Chroma Codes (SGLang version) ```bash Command theme={null} git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git cd Chroma-SGLang ``` ### Step 4: Run the Server ```bash Command theme={null} docker run -d \ --gpus all \ -p 8000:8000 \ -w /app/Chroma-SGLang \ -v "your_Chroma-SGLang_path":/app/Chroma-SGLang \ -v "your_chroma_path":/model \ -e CHROMA_MODEL_PATH=/model \ -e DP_SIZE="1" \ flashlabs/chroma:latest \ /opt/conda/bin/python -m uvicorn api_server:app \ --host 0.0.0.0 \ --port 8000 \ --workers 1 ``` or run simply the following one line command ```bash Command theme={null} docker-compose up -d ``` ## 5. Client Usage Example Once the server is running, you can interact with it using HTTP requests. ### Python Client ```python Example theme={null} import requests import base64 url = "http://localhost:8000/v1/chat/completions" headers = {"Content-Type": "application/json"} payload = { "model": "chroma", "messages": [ { "role": "system", "content": "You are Chroma, a voice agent developed by FlashLabs." }, { "role": "user", "content": [ {"type": "audio", "audio": "assets/question_audio.wav"} ] } ], "max_tokens": 1000, "return_audio": True } response = requests.post(url, json=payload, headers=headers) result = response.json() if result.get("audio"): audio_data = base64.b64decode(result["audio"]) with open("output.wav", "wb") as f: f.write(audio_data) print("Audio saved to output.wav") ``` ### OpenAI SDK Compatible Example ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="dummy", base_url="http://localhost:8000/v1" ) response = client.chat.completions.create( model="chroma", messages=[ {"role": "system", "content": "You are a helpful assistant."}, { "role": "user", "content": [ {"type": "audio", "audio": "assets/question_audio.wav"} ] } ], extra_body={ "prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.", "prompt_audio": "assets/ref_audio.wav", "return_audio": True } ) print(response) ``` ### CLI (cURL) ```bash Command theme={null} curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "chroma", "messages": [ { "role": "system", "content": "You are Chroma, a voice agent developed by FlashLabs." }, { "role": "user", "content": [ { "type": "audio", "audio": "assets/question_audio.wav" } ] } ], "max_tokens": 1000, "return_audio": true }' | jq -r '.audio' | base64 -d > output.wav ``` # GLM-4.5 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5 ## 1. Model Introduction [GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X * **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X **License:** Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.5 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.5", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x) * Model: GLM-4.5 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.5 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-questions 200 \ --port 30000 ``` # GLM-4.5V Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5V ## 1. Model Introduction [GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types. **Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X GLM-4.5V introduces several key features: * **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection. * **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences. * **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces. * **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context. * **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command. ### 3.2 Configuration Tips * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`. * **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`. * **Hardware Notes:** * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. * **Additional Multimodal Parameters:** * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). * `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies. * `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --mm-enable-dp-encoder \ --enable-metrics ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs GLM-4.5V supports both image and video inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Describe this image in detail." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.37s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** GLM-4.5V can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.86s Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot). ``` **Video Input Example:** GLM-4.5V supports video understanding by processing video URLs: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Note:** * For video processing, ensure you have sufficient context length configured (up to 64K tokens) * Video processing may require more memory; adjust `--mem-fraction-static` accordingly * You can also provide local file paths using `file://` protocol **Example Output:** ```text Output theme={null} Response costs: 3.89s Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen. ``` #### 4.2.2 Thinking Mode GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --reasoning-parser glm45 \ --tp 4 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. **Disable Thinking Mode:** To disable thinking mode for a specific request: ```python Example theme={null} response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[{"role": "user", "content": "What is the capital of France?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) ``` #### 4.2.3 Tool Calling GLM-4.5V supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/GLM-4.5V \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 4 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.4 Thinking Budget Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.5V", messages=[{"role": "user", "content": "Describe this image briefly."}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.1.1 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 ``` * Test Result ```text Output theme={null} Benchmark time: 616.6163094160147 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.867, 'num': 30}, 'Agriculture': {'acc': 0.567, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.667, 'num': 30}, 'Art': {'acc': 0.667, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.8, 'num': 30}, 'Biology': {'acc': 0.6, 'num': 30}, 'Chemistry': {'acc': 0.533, 'num': 30}, 'Clinical_Medicine': {'acc': 0.667, 'num': 30}, 'Computer_Science': {'acc': 0.8, 'num': 30}, 'Design': {'acc': 0.867, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30}, 'Economics': {'acc': 0.833, 'num': 30}, 'Electronics': {'acc': 0.433, 'num': 30}, 'Energy_and_Power': {'acc': 0.733, 'num': 30}, 'Finance': {'acc': 0.767, 'num': 30}, 'Geography': {'acc': 0.667, 'num': 30}, 'History': {'acc': 0.8, 'num': 30}, 'Literature': {'acc': 0.9, 'num': 30}, 'Manage': {'acc': 0.733, 'num': 30}, 'Marketing': {'acc': 0.9, 'num': 30}, 'Materials': {'acc': 0.567, 'num': 30}, 'Math': {'acc': 0.8, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.767, 'num': 30}, 'Music': {'acc': 0.3, 'num': 30}, 'Overall': {'acc': 0.732, 'num': 900}, 'Overall-Art and Design': {'acc': 0.683, 'num': 120}, 'Overall-Business': {'acc': 0.82, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.787, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120}, 'Overall-Science': {'acc': 0.707, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210}, 'Pharmacy': {'acc': 0.9, 'num': 30}, 'Physics': {'acc': 0.933, 'num': 30}, 'Psychology': {'acc': 0.767, 'num': 30}, 'Public_Health': {'acc': 0.9, 'num': 30}, 'Sociology': {'acc': 0.667, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.732 ``` # GLM-4.6 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6 ## 1. Model Introduction [GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows: * **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks. * **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages. * **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability. * **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks. * **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios. For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.6 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.6", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x) * Model: GLM-4.6 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 63.82 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4209 Request throughput (req/s): 0.16 Input token throughput (tok/s): 95.60 Output token throughput (tok/s): 65.97 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 161.57 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6379.24 Median E2E Latency (ms): 5085.00 ---------------Time to First Token---------------- Mean TTFT (ms): 155.57 Median TTFT (ms): 149.79 P99 TTFT (ms): 207.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.81 Median TPOT (ms): 14.80 P99 TPOT (ms): 14.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.82 Median ITL (ms): 14.82 P95 ITL (ms): 15.17 P99 ITL (ms): 15.36 Max ITL (ms): 25.05 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 72.06 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40672 Request throughput (req/s): 1.11 Input token throughput (tok/s): 550.47 Output token throughput (tok/s): 565.14 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1115.61 Concurrency: 13.71 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12348.93 Median E2E Latency (ms): 13164.81 ---------------Time to First Token---------------- Mean TTFT (ms): 196.08 Median TTFT (ms): 155.22 P99 TTFT (ms): 377.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.24 Median TPOT (ms): 24.55 P99 TPOT (ms): 30.42 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.92 Median ITL (ms): 21.40 P95 ITL (ms): 22.49 P99 ITL (ms): 123.83 Max ITL (ms): 486.54 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 138.50 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251841 Request throughput (req/s): 3.61 Input token throughput (tok/s): 1803.78 Output token throughput (tok/s): 1820.61 Peak output token throughput (tok/s): 2900.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3624.40 Concurrency: 90.91 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25183.97 Median E2E Latency (ms): 23968.49 ---------------Time to First Token---------------- Mean TTFT (ms): 337.77 Median TTFT (ms): 180.65 P99 TTFT (ms): 906.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 49.97 Median TPOT (ms): 52.20 P99 TPOT (ms): 61.81 ---------------Inter-Token Latency---------------- Mean ITL (ms): 49.36 Median ITL (ms): 35.05 P95 ITL (ms): 124.91 P99 ITL (ms): 187.69 Max ITL (ms): 440.34 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 666.64 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44387 Request throughput (req/s): 0.02 Input token throughput (tok/s): 9.15 Output token throughput (tok/s): 66.68 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 75.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 66661.35 Median E2E Latency (ms): 71902.36 ---------------Time to First Token---------------- Mean TTFT (ms): 160.21 Median TTFT (ms): 140.32 P99 TTFT (ms): 295.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.92 Median TPOT (ms): 14.94 P99 TPOT (ms): 15.02 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.96 Median ITL (ms): 14.96 P95 ITL (ms): 15.36 P99 ITL (ms): 15.57 Max ITL (ms): 19.06 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 503.30 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 318025 Request throughput (req/s): 0.16 Input token throughput (tok/s): 78.82 Output token throughput (tok/s): 632.28 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 711.09 Concurrency: 13.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 87349.22 Median E2E Latency (ms): 88248.04 ---------------Time to First Token---------------- Mean TTFT (ms): 228.54 Median TTFT (ms): 142.78 P99 TTFT (ms): 569.84 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 21.97 Median TPOT (ms): 22.14 P99 TPOT (ms): 22.47 ---------------Inter-Token Latency---------------- Mean ITL (ms): 21.91 Median ITL (ms): 21.80 P95 ITL (ms): 22.30 P99 ITL (ms): 22.78 Max ITL (ms): 137.19 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 772.28 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1299924 Request throughput (req/s): 0.41 Input token throughput (tok/s): 205.80 Output token throughput (tok/s): 1684.24 Peak output token throughput (tok/s): 2112.00 Peak concurrent requests: 68 Total token throughput (tok/s): 1890.05 Concurrency: 56.17 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 135563.36 Median E2E Latency (ms): 140888.88 ---------------Time to First Token---------------- Mean TTFT (ms): 232.45 Median TTFT (ms): 145.59 P99 TTFT (ms): 576.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.47 Median TPOT (ms): 34.02 P99 TPOT (ms): 35.10 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.30 Median ITL (ms): 32.63 P95 ITL (ms): 34.27 P99 ITL (ms): 104.39 Max ITL (ms): 155.65 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 65.11 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4210 Request throughput (req/s): 0.15 Input token throughput (tok/s): 644.17 Output token throughput (tok/s): 64.66 Peak output token throughput (tok/s): 68.00 Peak concurrent requests: 2 Total token throughput (tok/s): 708.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6508.31 Median E2E Latency (ms): 5263.36 ---------------Time to First Token---------------- Mean TTFT (ms): 189.48 Median TTFT (ms): 159.23 P99 TTFT (ms): 304.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 15.02 Median TPOT (ms): 15.03 P99 TPOT (ms): 15.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.04 Median ITL (ms): 15.03 P95 ITL (ms): 15.46 P99 ITL (ms): 15.65 Max ITL (ms): 24.20 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 76.43 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41577 Request throughput (req/s): 1.05 Input token throughput (tok/s): 3925.47 Output token throughput (tok/s): 544.15 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 4469.62 Concurrency: 13.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13329.63 Median E2E Latency (ms): 14141.09 ---------------Time to First Token---------------- Mean TTFT (ms): 339.88 Median TTFT (ms): 252.75 P99 TTFT (ms): 906.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.37 Median TPOT (ms): 25.73 P99 TPOT (ms): 30.94 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.04 Median ITL (ms): 21.68 P95 ITL (ms): 22.69 P99 ITL (ms): 146.98 Max ITL (ms): 483.14 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 136.24 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169452 Request throughput (req/s): 2.35 Input token throughput (tok/s): 9350.32 Output token throughput (tok/s): 1245.44 Peak output token throughput (tok/s): 1984.00 Peak concurrent requests: 69 Total token throughput (tok/s): 10595.77 Concurrency: 58.46 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 24889.40 Median E2E Latency (ms): 25123.37 ---------------Time to First Token---------------- Mean TTFT (ms): 355.82 Median TTFT (ms): 268.84 P99 TTFT (ms): 858.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 46.62 Median TPOT (ms): 49.04 P99 TPOT (ms): 58.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 46.36 Median ITL (ms): 32.46 P95 ITL (ms): 135.23 P99 ITL (ms): 204.27 Max ITL (ms): 508.14 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-questions 200 \ --port 30000 ``` * Test Result ```text Output theme={null} Accuracy: 0.975 Invalid: 0.000 Latency: 16.574 s Output throughput: 1194.637 token/s ``` # GLM-4.6V Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6V ## 1. Model Introduction GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios. Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features: * **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling). * **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content. * **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text. * **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. ### 2.1 Docker Installation (Recommended) ```shell Command theme={null} docker pull lmsysorg/sglang:latest ``` **Advantages:** * Ready to use out of the box, no manual environment configuration needed * Avoids dependency conflict issues * Easy to migrate between different environments ### 2.2 Build from Source If you need to use the latest development version or require custom modifications, you can build from source: ```bash Command theme={null} # Install SGLang using UV (recommended) git clone https://github.com/sgl-project/sglang.git cd sglang uv venv source .venv/bin/activate uv pip install --prerelease=allow -e "python[all]" --index-url=https://pypi.org/simple pip install nvidia-cudnn-cu12==9.16.0.29 # Install ffmpeg to support video input sudo apt update sudo apt install ffmpeg ``` **Use Cases:** * Need to customize and modify SGLang source code * Want to use the latest development features * Participate in SGLang project development For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command. ### 3.2 Configuration Tips * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically). * **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`. * **Hardware Notes:** * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. * **Additional Multimodal Parameters:** * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). * `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies. * `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path zai-org/GLM-4.6V \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --mm-enable-dp-encoder \ --enable-metrics ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs GLM-4.6V supports image and video inputs via the OpenAI-compatible API. **Image Input:** ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{30000}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "default", "messages": [ {{ "role": "user", "content": [ {{ "type": "image_url", "image_url": {{ "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true" }} }}, {{ "type": "text", "text": "What is the image" }} ] }} ], "temperature": "0", "max_completion_tokens": "1000", "max_tokens": "1000" }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print(response) ``` ```text Output theme={null} {"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} ``` **Video Input:** ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{30000}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "default", "messages": [ {{ "role": "user", "content": [ {{ "type": "video_url", "video_url": {{ "url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4" }} }}, {{ "type": "text", "text": "What is in the video" }} ] }} ], "temperature": "0", "max_completion_tokens": "1000", "max_tokens": "1000" }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print(response) ``` ```text Output theme={null} {"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n* **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n* **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n* **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n* **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}} ``` #### 4.2.2 Thinking Mode GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.6V \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request: ```python Example theme={null} from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://127.0.0.1:30000/v1" client = OpenAI(api_key=openai_api_key, base_url=openai_api_base) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get current temperature for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and country e.g. Beijing, China", } }, "required": ["location"], "additionalProperties": False, }, }, } ] messages = [ { "role": "user", "content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image." }, { "role": "assistant", "tool_calls": [ { "id": "call_bk32t88BGpSdbtDgzT044Rh4", "type": "function", "function": { "name": 'get_weather', "arguments": '{"location":"Beijing, China"}' } } ] }, { "role": "tool", "tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4", "content": [ { "type": "text", "text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C." }, { "type": "image_url", "image_url": { "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true" } } ] }, ] response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=messages, timeout=900, tools=tools ) print(response.choices[0].message.content.strip()) ``` **Output Example:** ```text Output theme={null} The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C. Yes, the tool returned an image (the SGL logo). ``` #### 4.2.4 Thinking Budget Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget): ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.6V", messages=[{"role": "user", "content": "Describe this image briefly."}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark ### 5.1. Text Benchmark: Latency, Throughput and Accuracy #### Command ```shell Command theme={null} python3 ./benchmark/gsm8k/bench_sglang.py ``` #### Result Output ```text Output theme={null} Accuracy: 0.925 Invalid: 0.000 Latency: 15.327 s Output throughput: 1788.375 token/s ``` ### 5.2. Multimodal Benchmark - Latency and Throughput #### Command ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --port 30000 \ --model zai-org/GLM-4.6V \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 128 \ --max-concurrency 8 ``` #### Result Output ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 8 Successful requests: 128 Benchmark duration (s): 89.27 Total input tokens: 315390 Total input text tokens: 8702 Total input vision tokens: 306688 Total generated tokens: 66020 Total generated tokens (retokenized): 31037 Request throughput (req/s): 1.43 Input token throughput (tok/s): 3533.17 Output token throughput (tok/s): 739.59 Peak output token throughput (tok/s): 823.00 Peak concurrent requests: 12 Total token throughput (tok/s): 4272.76 Concurrency: 7.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5349.20 Median E2E Latency (ms): 5380.98 ---------------Time to First Token---------------- Mean TTFT (ms): 1724.04 Median TTFT (ms): 1688.16 P99 TTFT (ms): 6152.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.15 Median TPOT (ms): 7.77 P99 TPOT (ms): 23.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.00 Median ITL (ms): 8.44 P95 ITL (ms): 9.23 P99 ITL (ms): 116.02 Max ITL (ms): 173.48 ================================================== ``` ### 5.3. Multimodal Accuracy Benchmark - MMMU #### Command ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}' ``` #### Result Output ```text Output theme={null} Benchmark time: 487.2229107860476 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.962, 'num': 26}, 'Agriculture': {'acc': 0.5, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.733, 'num': 15}, 'Art': {'acc': 0.833, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.733, 'num': 30}, 'Biology': {'acc': 0.586, 'num': 29}, 'Chemistry': {'acc': 0.654, 'num': 26}, 'Clinical_Medicine': {'acc': 0.633, 'num': 30}, 'Computer_Science': {'acc': 0.76, 'num': 25}, 'Design': {'acc': 0.867, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30}, 'Economics': {'acc': 0.862, 'num': 29}, 'Electronics': {'acc': 0.5, 'num': 18}, 'Energy_and_Power': {'acc': 0.875, 'num': 16}, 'Finance': {'acc': 0.857, 'num': 28}, 'Geography': {'acc': 0.714, 'num': 28}, 'History': {'acc': 0.767, 'num': 30}, 'Literature': {'acc': 0.897, 'num': 29}, 'Manage': {'acc': 0.759, 'num': 29}, 'Marketing': {'acc': 1.0, 'num': 26}, 'Materials': {'acc': 0.833, 'num': 18}, 'Math': {'acc': 0.76, 'num': 25}, 'Mechanical_Engineering': {'acc': 0.619, 'num': 21}, 'Music': {'acc': 0.286, 'num': 28}, 'Overall': {'acc': 0.761, 'num': 803}, 'Overall-Art and Design': {'acc': 0.729, 'num': 118}, 'Overall-Business': {'acc': 0.884, 'num': 138}, 'Overall-Health and Medicine': {'acc': 0.773, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118}, 'Overall-Science': {'acc': 0.728, 'num': 136}, 'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143}, 'Pharmacy': {'acc': 0.933, 'num': 30}, 'Physics': {'acc': 0.929, 'num': 28}, 'Psychology': {'acc': 0.733, 'num': 30}, 'Public_Health': {'acc': 0.933, 'num': 30}, 'Sociology': {'acc': 0.724, 'num': 29}} eval out saved to ./val_sglang.json Overall accuracy: 0.761 ``` # GLM-4.7 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7 ## 1. Model Introduction [GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows. GLM-4.7 brings improvements across all major domains: * **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations * **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing * **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance * **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows * **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7). **Key Features:** * **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks * **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs * **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) * **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8) * **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4) **License:** Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA H100 / H200 / B200 `lmsysorg/sglang:v0.5.12`
NVIDIA GB200 / B300 / GB300 (aarch64) `lmsysorg/sglang:v0.5.12-cu130`
AMD MI300X / MI325X `lmsysorg/sglang:v0.5.12-rocm720-mi30x`
AMD MI355X `lmsysorg/sglang:v0.5.12-rocm720-mi35x`
## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:
Hardware NVFP4 FP8 BF16
B200 (8×, single node) tp=2 / 4 / 8 tp=4 / 8 tp=8
GB200 (NVL72, 4× per tray) tp=2 / 4 tp=4 —
H200 (8×) — tp=8 tp=8
AMD MI300X / MI325X / MI355X — tp=2 / 4 / 8 tp=4 / 8
* **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above. * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3). For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --reasoning-parser glm45 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.7 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Thinking Budget Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`: ```python Example theme={null} import openai from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*") response = client.chat.completions.create( model="zai-org/GLM-4.7", messages=[{"role": "user", "content": "Is Paris the Capital of France?"}], max_tokens=1024, extra_body={ "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(), "custom_params": {"thinking_budget": 512}, }, ) print(response) ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x) * Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD * SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD) * Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config. **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Four core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
**Throughput** 4K 1K Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below. #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7 \ --tp 8 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` **Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4** The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1. > **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks. #### 5.1.4.1 NVIDIA B200 **Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):** ```bash Command theme={null} python -m sglang.launch_server \ --model nvidia/GLM-4.7-NVFP4 \ --tp-size 2 \ --mem-fraction-static 0.85 \ --reasoning-parser glm45 \ --tool-call-parser glm47 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 5 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 5 Benchmark duration (s): 25.07 Total input tokens: 8105 Total generated tokens: 2674 Request throughput (req/s): 0.20 Input token throughput (tok/s): 323.25 Output token throughput (tok/s): 106.65 Total token throughput (tok/s): 429.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5011.93 Median E2E Latency (ms): 6441.44 ---------------Time to First Token---------------- Mean TTFT (ms): 179.61 Median TTFT (ms): 169.05 P99 TTFT (ms): 238.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.05 Median TPOT (ms): 9.03 P99 TPOT (ms): 9.16 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.05 Median ITL (ms): 9.05 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.60 Total input tokens: 179772 Total generated tokens: 39657 Request throughput (req/s): 1.32 Input token throughput (tok/s): 2966.39 Output token throughput (tok/s): 654.37 Total token throughput (tok/s): 3620.76 Concurrency: 14.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10615.87 Median E2E Latency (ms): 9985.45 ---------------Time to First Token---------------- Mean TTFT (ms): 267.39 Median TTFT (ms): 177.26 P99 TTFT (ms): 584.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.98 Median TPOT (ms): 21.06 P99 TPOT (ms): 24.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.92 Median ITL (ms): 17.93 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 640 \ --max-concurrency 128 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 128 Successful requests: 640 Benchmark duration (s): 172.95 Total input tokens: 1453591 Total generated tokens: 308740 Request throughput (req/s): 3.70 Input token throughput (tok/s): 8404.67 Output token throughput (tok/s): 1785.14 Total token throughput (tok/s): 10189.80 Concurrency: 117.85 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 31848.20 Median E2E Latency (ms): 28554.42 ---------------Time to First Token---------------- Mean TTFT (ms): 1598.40 Median TTFT (ms): 298.88 P99 TTFT (ms): 11015.96 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 65.94 Median TPOT (ms): 65.81 P99 TPOT (ms): 137.73 ---------------Inter-Token Latency---------------- Mean ITL (ms): 62.99 Median ITL (ms): 35.44 ================================================== ``` #### 5.1.4.2 NVIDIA GB200 **Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):** ```bash Command theme={null} python -m sglang.launch_server \ --model nvidia/GLM-4.7-NVFP4 \ --tp-size 2 \ --mem-fraction-static 0.85 \ --reasoning-parser glm45 \ --tool-call-parser glm47 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 5 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 5 Benchmark duration (s): 24.74 Total input tokens: 8105 Total generated tokens: 2674 Request throughput (req/s): 0.20 Input token throughput (tok/s): 327.65 Output token throughput (tok/s): 108.10 Total token throughput (tok/s): 435.75 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4944.47 Median E2E Latency (ms): 6347.31 ---------------Time to First Token---------------- Mean TTFT (ms): 211.41 Median TTFT (ms): 207.25 P99 TTFT (ms): 226.46 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.86 Median TPOT (ms): 8.84 P99 TPOT (ms): 8.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.87 Median ITL (ms): 8.85 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.40 Total input tokens: 179772 Total generated tokens: 39657 Request throughput (req/s): 1.32 Input token throughput (tok/s): 2976.52 Output token throughput (tok/s): 656.61 Total token throughput (tok/s): 3633.13 Concurrency: 13.97 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10611.51 Median E2E Latency (ms): 9956.84 ---------------Time to First Token---------------- Mean TTFT (ms): 338.14 Median TTFT (ms): 215.25 P99 TTFT (ms): 915.40 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.87 Median TPOT (ms): 21.36 P99 TPOT (ms): 27.05 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.77 Median ITL (ms): 16.53 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model nvidia/GLM-4.7-NVFP4 \ --dataset-name random \ --random-input-len 4096 \ --random-output-len 1024 \ --num-prompts 640 \ --max-concurrency 128 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 128 Successful requests: 640 Benchmark duration (s): 181.89 Total input tokens: 1453591 Total generated tokens: 309221 Request throughput (req/s): 3.52 Input token throughput (tok/s): 7991.59 Output token throughput (tok/s): 1700.04 Total token throughput (tok/s): 9691.63 Concurrency: 118.86 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 33690.47 Median E2E Latency (ms): 30421.55 ---------------Time to First Token---------------- Mean TTFT (ms): 1353.16 Median TTFT (ms): 383.52 P99 TTFT (ms): 8940.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 69.88 Median TPOT (ms): 71.77 P99 TPOT (ms): 131.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 67.23 Median ITL (ms): 33.46 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-shots 5 \ --num-questions 1319 \ --port 30000 ``` * Test Result (NVIDIA B200, TP=2 NVFP4) ```text Output theme={null} Accuracy: 0.946 Latency: 178.284 s Output throughput: 769.204 token/s ``` * Test Result (NVIDIA GB200, TP=2 NVFP4) ```text Output theme={null} Accuracy: 0.951 Latency: 175.190 s Invalid: 0.000 ``` # GLM-4.7-Flash Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7-Flash ## 1. Model Introduction [GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment. As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency: * **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference * **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance * **Superior Coding**: Strong code generation and understanding capabilities * **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows * **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7). **Key Features:** * **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) **License:** Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --reasoning-parser glm45 \ --attention-backend triton \ --tp 1 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation. GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --attention-backend triton \ --tp 1 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls (tool call deltas may stream in multiple chunks) if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls if tool_calls_accumulator: print("\n=============== Tool Calls =================", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you. =============== Tool Calls ================= Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/GLM-4.7-Flash", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 (1x) * Model: GLM-4.7-Flash * Tensor Parallelism: 1 * SGLang Version: 0.5.7 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Test Scenarios Three core scenarios reflect real-world usage patterns:
Scenario Input Length Output Length Use Case
**Chat** 1K 1K Most common conversational AI workload
**Reasoning** 1K 8K Long-form generation, complex reasoning tasks
**Summarization** 8K 1K Document summarization, RAG retrieval
#### 5.1.2 Concurrency Levels Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier): * **Low Concurrency**: `--max-concurrency 1` (Latency-optimized) * **Medium Concurrency**: `--max-concurrency 16` (Balanced) * **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) #### 5.1.3 Number of Prompts For each concurrency level, configure `num_prompts` to simulate realistic user loads: * **Quick Test**: `num_prompts = concurrency × 1` (minimal test) * **Recommended**: `num_prompts = concurrency × 5` (standard benchmark) * **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade) *** #### 5.1.4 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/GLM-4.7-Flash \ --attention-backend triton \ --tp 1 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 38.94 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.26 Input token throughput (tok/s): 156.67 Output token throughput (tok/s): 108.37 Peak output token throughput (tok/s): 125.00 Peak concurrent requests: 2 Total token throughput (tok/s): 265.03 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3891.12 Median E2E Latency (ms): 3061.48 P90 E2E Latency (ms): 7172.25 P99 E2E Latency (ms): 9042.62 ---------------Time to First Token---------------- Mean TTFT (ms): 131.36 Median TTFT (ms): 94.55 P99 TTFT (ms): 435.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.75 Median TPOT (ms): 8.82 P99 TPOT (ms): 9.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.93 Median ITL (ms): 8.98 P95 ITL (ms): 9.83 P99 ITL (ms): 10.20 Max ITL (ms): 18.50 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 52.73 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 1.52 Input token throughput (tok/s): 752.27 Output token throughput (tok/s): 773.83 Peak output token throughput (tok/s): 1040.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1526.10 Concurrency: 13.98 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9217.90 Median E2E Latency (ms): 9642.50 P90 E2E Latency (ms): 15147.02 P99 E2E Latency (ms): 18237.06 ---------------Time to First Token---------------- Mean TTFT (ms): 299.02 Median TTFT (ms): 105.98 P99 TTFT (ms): 1109.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.03 Median TPOT (ms): 18.00 P99 TPOT (ms): 26.51 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.52 Median ITL (ms): 16.07 P95 ITL (ms): 18.14 P99 ITL (ms): 89.43 Max ITL (ms): 763.13 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 91.48 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 250941 Request throughput (req/s): 5.47 Input token throughput (tok/s): 2730.87 Output token throughput (tok/s): 2761.82 Peak output token throughput (tok/s): 4199.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5492.69 Concurrency: 90.54 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16566.04 Median E2E Latency (ms): 16134.36 P90 E2E Latency (ms): 30167.60 P99 E2E Latency (ms): 34034.04 ---------------Time to First Token---------------- Mean TTFT (ms): 433.94 Median TTFT (ms): 123.26 P99 TTFT (ms): 1760.09 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.26 Median TPOT (ms): 33.56 P99 TPOT (ms): 38.78 ---------------Inter-Token Latency---------------- Mean ITL (ms): 31.99 Median ITL (ms): 24.06 P95 ITL (ms): 79.62 P99 ITL (ms): 103.03 Max ITL (ms): 1369.20 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 525.43 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44451 Request throughput (req/s): 0.02 Input token throughput (tok/s): 11.61 Output token throughput (tok/s): 84.62 Peak output token throughput (tok/s): 125.00 Peak concurrent requests: 2 Total token throughput (tok/s): 96.23 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 52540.19 Median E2E Latency (ms): 53694.45 P90 E2E Latency (ms): 94742.08 P99 E2E Latency (ms): 101224.18 ---------------Time to First Token---------------- Mean TTFT (ms): 97.45 Median TTFT (ms): 95.28 P99 TTFT (ms): 105.64 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.94 Median TPOT (ms): 11.25 P99 TPOT (ms): 13.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.80 Median ITL (ms): 11.51 P95 ITL (ms): 15.83 P99 ITL (ms): 16.86 Max ITL (ms): 19.96 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 473.92 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 317860 Request throughput (req/s): 0.17 Input token throughput (tok/s): 83.70 Output token throughput (tok/s): 671.65 Peak output token throughput (tok/s): 1040.00 Peak concurrent requests: 19 Total token throughput (tok/s): 755.35 Concurrency: 13.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 81746.73 Median E2E Latency (ms): 78508.54 P90 E2E Latency (ms): 155292.49 P99 E2E Latency (ms): 166769.99 ---------------Time to First Token---------------- Mean TTFT (ms): 117.50 Median TTFT (ms): 101.97 P99 TTFT (ms): 182.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.36 Median TPOT (ms): 20.48 P99 TPOT (ms): 22.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.52 Median ITL (ms): 20.42 P95 ITL (ms): 23.41 P99 ITL (ms): 26.29 Max ITL (ms): 90.48 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 714.72 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 1301025 Total generated tokens (retokenized): 1289431 Request throughput (req/s): 0.45 Input token throughput (tok/s): 222.38 Output token throughput (tok/s): 1820.33 Peak output token throughput (tok/s): 3200.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2042.71 Concurrency: 55.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 124364.58 Median E2E Latency (ms): 129250.98 P90 E2E Latency (ms): 219175.80 P99 E2E Latency (ms): 247741.77 ---------------Time to First Token---------------- Mean TTFT (ms): 149.40 Median TTFT (ms): 114.78 P99 TTFT (ms): 288.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 30.51 Median TPOT (ms): 31.75 P99 TPOT (ms): 33.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 30.56 Median ITL (ms): 30.82 P95 ITL (ms): 33.20 P99 ITL (ms): 80.54 Max ITL (ms): 117.72 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 58.27 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.17 Input token throughput (tok/s): 719.73 Output token throughput (tok/s): 72.42 Peak output token throughput (tok/s): 112.00 Peak concurrent requests: 2 Total token throughput (tok/s): 792.15 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5825.08 Median E2E Latency (ms): 4624.26 P90 E2E Latency (ms): 12690.22 P99 E2E Latency (ms): 13177.96 ---------------Time to First Token---------------- Mean TTFT (ms): 296.01 Median TTFT (ms): 195.59 P99 TTFT (ms): 717.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.63 Median TPOT (ms): 13.07 P99 TPOT (ms): 16.68 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.13 Median ITL (ms): 13.17 P95 ITL (ms): 17.02 P99 ITL (ms): 17.47 Max ITL (ms): 19.84 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 89.59 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41656 Request throughput (req/s): 0.89 Input token throughput (tok/s): 3348.77 Output token throughput (tok/s): 465.10 Peak output token throughput (tok/s): 752.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3813.87 Concurrency: 14.39 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16120.74 Median E2E Latency (ms): 16246.55 P90 E2E Latency (ms): 27279.72 P99 E2E Latency (ms): 34577.93 ---------------Time to First Token---------------- Mean TTFT (ms): 1943.94 Median TTFT (ms): 382.19 P99 TTFT (ms): 8980.41 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 27.87 Median TPOT (ms): 28.26 P99 TPOT (ms): 40.55 ---------------Inter-Token Latency---------------- Mean ITL (ms): 27.27 Median ITL (ms): 21.74 P95 ITL (ms): 23.32 P99 ITL (ms): 232.65 Max ITL (ms): 4282.01 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-4.7-Flash \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 167.01 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169226 Request throughput (req/s): 1.92 Input token throughput (tok/s): 7627.82 Output token throughput (tok/s): 1017.93 Peak output token throughput (tok/s): 1984.00 Peak concurrent requests: 69 Total token throughput (tok/s): 8645.75 Concurrency: 59.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 31147.52 Median E2E Latency (ms): 30603.34 P90 E2E Latency (ms): 54889.44 P99 E2E Latency (ms): 67665.30 ---------------Time to First Token---------------- Mean TTFT (ms): 428.87 Median TTFT (ms): 441.69 P99 TTFT (ms): 1232.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 58.06 Median TPOT (ms): 62.79 P99 TPOT (ms): 82.23 ---------------Inter-Token Latency---------------- Mean ITL (ms): 57.93 Median ITL (ms): 33.30 P95 ITL (ms): 247.98 P99 ITL (ms): 409.63 Max ITL (ms): 1421.21 ================================================== ``` #### 5.1.5 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-questions 200 \ --port 30000 ``` * Result ```text Output theme={null} Accuracy: 0.845 Invalid: 0.000 Latency: 8.431 s Output throughput: 2195.387 token/s ``` # GLM-5 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5 ## 1. Model Introduction [GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity. With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks. **Key Features:** * **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks * **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2) * **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5) * **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8) **License:** MIT ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.
Hardware FP8 BF16
H100 tp=16 tp=32
H200 tp=8 tp=16
B200 tp=8 tp=16
MI300X/MI325X — tp=8
MI355X — tp=8
* **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization. * **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5. * For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. * Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature. **FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off. ## 4. Model Invocation Deploy GLM-5 with the following command (FP8 on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5-FP8 \ --tp 8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-flashinfer-allreduce-fusion \ --mem-fraction-static 0.85 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above. ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5 \ --tp 8 \ --trust-remote-code \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants me to solve a math problem: "What is 15% of 240?". Step 1: Understand the problem. I need to calculate a percentage of a number. Formula: Percentage × Number = Result. Step 2: Convert the percentage to a decimal or fraction. 15% = 15/100 or 0.15. Step 3: Perform the multiplication. Method A: Decimal multiplication. 0.15 × 240. Break it down: 10% of 240 = 24. 5% is half of 10%, so 12. 15% = 10% + 5% = 24 + 12 = 36. Method B: Fraction multiplication. 15/100 × 240. Simplify 240/100 = 2.4. 15 × 2.4. 10 × 2.4 = 24. 5 × 2.4 = 12. 24 + 12 = 36. Method C: Direct multiplication. 240 × 0.15. 240 × 0.10 = 24. 240 × 0.05 = 12. 24 + 12 = 36. Step 4: Final Verification. Is 36 reasonable? 10% is 24. 20% is 48. 15% is halfway between 10% and 20%. Halfway between 24 and 48 is 36. The result is correct. Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%). =============== Content ================= Here is the step-by-step solution: **Step 1: Convert the percentage to a decimal.** To convert 15% to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ **Step 2: Multiply the decimal by the number.** Now, multiply 0.15 by 240. $$0.15 \times 240$$ **Step 3: Perform the calculation.** You can break this down to make it easier: $$0.15 = 0.10 + 0.05$$ * First, find 10% of 240: $$0.10 \times 240 = 24$$ * Next, find 5% (which is half of 10%): $$\frac{24}{2} = 12$$ * Add the two results together: $$24 + 12 = 36$$ **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find **15% of 240**, follow these steps: ### Step 1: Convert the Percentage to a Decimal First, convert the percentage to a decimal by dividing by 100. \[ 15\% = \frac{15}{100} = 0.15 \] ### Step 2: Multiply by the Number Next, multiply the decimal by the number you want to find the percentage of. \[ 0.15 \times 240 \] ### Step 3: Perform the Multiplication Calculate the multiplication: \[ 0.15 \times 240 = 36 \] ### Final Answer \[ \boxed{36} \] ``` #### 4.3.2 Tool Calling GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-5-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required: - location: required, should be "Beijing" - unit: optional (not in required array), can be "celsius" or "fahrenheit" Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Be Tool Call: None Arguments: ijing" Tool Call: None Arguments: } ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: GLM-5-FP8 * Tensor Parallelism: 8 * SGLang Version: commit 947927bdb #### 5.1.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.78 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4213 Request throughput (req/s): 0.28 Input token throughput (tok/s): 170.54 Output token throughput (tok/s): 117.96 Peak output token throughput (tok/s): 148.00 Peak concurrent requests: 2 Total token throughput (tok/s): 288.50 Concurrency: 1.00 Accept length: 3.48 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3576.31 Median E2E Latency (ms): 2935.97 P90 E2E Latency (ms): 5908.97 P99 E2E Latency (ms): 8588.08 ---------------Time to First Token---------------- Mean TTFT (ms): 290.88 Median TTFT (ms): 282.34 P99 TTFT (ms): 332.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.54 Median TPOT (ms): 6.97 P99 TPOT (ms): 9.04 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.80 Median ITL (ms): 6.81 P95 ITL (ms): 13.51 P99 ITL (ms): 26.99 Max ITL (ms): 29.50 ================================================== ``` #### 5.1.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 411.74 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 499614 Request throughput (req/s): 2.43 Input token throughput (tok/s): 1220.41 Output token throughput (tok/s): 1214.97 Peak output token throughput (tok/s): 2648.00 Peak concurrent requests: 105 Total token throughput (tok/s): 2435.38 Concurrency: 96.30 Accept length: 3.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 39648.76 Median E2E Latency (ms): 39058.12 P90 E2E Latency (ms): 57009.82 P99 E2E Latency (ms): 68880.33 ---------------Time to First Token---------------- Mean TTFT (ms): 20613.80 Median TTFT (ms): 21429.21 P99 TTFT (ms): 29543.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.73 Median TPOT (ms): 36.52 P99 TPOT (ms): 67.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.13 Median ITL (ms): 16.57 P95 ITL (ms): 86.01 P99 ITL (ms): 164.88 Max ITL (ms): 1307.02 ================================================== ``` ### 5.2 Accuracy Benchmark The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned. #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.000 Latency: 32.470 s Output throughput: 642.044 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py --port 30000 ``` * Test Result ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.860 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.932 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.640 subject: college_computer_science, #q:100, acc: 0.900 subject: college_mathematics, #q:100, acc: 0.810 subject: college_medicine, #q:173, acc: 0.873 subject: college_physics, #q:102, acc: 0.912 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.928 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.897 subject: elementary_mathematics, #q:378, acc: 0.937 subject: formal_logic, #q:126, acc: 0.778 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.961 subject: high_school_chemistry, #q:203, acc: 0.847 subject: high_school_computer_science, #q:100, acc: 0.960 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.960 subject: high_school_government_and_politics, #q:193, acc: 0.984 subject: high_school_macroeconomics, #q:390, acc: 0.923 subject: high_school_mathematics, #q:270, acc: 0.696 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.821 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.889 subject: high_school_us_history, #q:204, acc: 0.941 subject: high_school_world_history, #q:237, acc: 0.945 subject: human_aging, #q:223, acc: 0.857 subject: human_sexuality, #q:131, acc: 0.908 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.907 subject: logical_fallacies, #q:163, acc: 0.933 subject: machine_learning, #q:112, acc: 0.830 subject: management, #q:103, acc: 0.942 subject: marketing, #q:234, acc: 0.940 subject: medical_genetics, #q:100, acc: 0.990 subject: miscellaneous, #q:783, acc: 0.959 subject: moral_disputes, #q:346, acc: 0.873 subject: moral_scenarios, #q:895, acc: 0.837 subject: nutrition, #q:306, acc: 0.922 subject: philosophy, #q:311, acc: 0.897 subject: prehistory, #q:324, acc: 0.929 subject: professional_accounting, #q:282, acc: 0.844 subject: professional_law, #q:1534, acc: 0.714 subject: professional_medicine, #q:272, acc: 0.941 subject: professional_psychology, #q:612, acc: 0.913 subject: public_relations, #q:110, acc: 0.791 subject: security_studies, #q:245, acc: 0.878 subject: sociology, #q:201, acc: 0.940 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.596 subject: world_religions, #q:171, acc: 0.936 Total latency: 165.275 Average accuracy: 0.877 ``` ### 5.3 AMD GPU Benchmarks #### 5.3.1 GSM8K Benchmark (MI325/MI35x) * MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends) ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 ``` ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 ``` Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911). # GLM-5.1 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.1 ## 1. Model Introduction **Available Models:** * **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1) * **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8) * **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4) **License:** MIT ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
Hardware NVFP4 FP8 BF16 MXFP4
H100 — tp=16 — —
H200 — tp=8 — —
B300 tp=8 — — —
GB300 tp=4 — — —
MI300X/MI325X — tp=8 tp=8 —
MI355X — tp=8 tp=8 tp=4
* **H100 and H200**: FP8 is the recommended deployment path. * **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300. * **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang. * For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. * Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature. ## 4. Model Invocation Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1-FP8 \ --tp 8 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.85 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 B300/GB300 (NVFP4) Server Command #### B300 ```shell Command theme={null} sglang serve \ --model-path nvidia/GLM-5.1-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --mem-fraction-static 0.80 \ --host 0.0.0.0 \ --port 30000 ``` #### GB300 ```shell Command theme={null} sglang serve \ --model-path nvidia/GLM-5.1-NVFP4 \ --tp 4 \ --quantization modelopt_fp4 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --mem-fraction-static 0.80 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above. #### MXFP4 (MI355X / gfx950) On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950. ```shell Command theme={null} # SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950 SGLANG_DSA_TRITON_PREFILL=1 sglang serve \ --model-path amd/GLM-5.1-MXFP4 \ --tp 4 \ --trust-remote-code \ --kv-cache-dtype fp8_e4m3 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.85 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` #### FP8 (Recommended) ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1-FP8 \ --tp 8 \ --trust-remote-code \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` #### BF16 ```shell Command theme={null} sglang serve \ --model-path zai-org/GLM-5.1 \ --tp 8 \ --trust-remote-code \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --chunked-prefill-size 131072 \ --mem-fraction-static 0.80 \ --watchdog-timeout 1200 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disable-custom-all-reduce \ --host 0.0.0.0 \ --port 30000 ``` ### 4.3 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.4 Advanced Usage #### 4.4.1 Reasoning Parser GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= 1. **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step. 2. **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$. 3. **Step-by-Step Breakdown:** * *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$. * *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$. * *Step 3: Perform the calculation.* * $0.15 \times 240$ * I can break this down further to make it easy to follow: * $0.10 \times 240 = 24$ (which is 10%) * $0.05 \times 240 = 12$ (which is 5%, half of 10%) * $24 + 12 = 36$ * Alternatively, standard multiplication: * $240 \times 15 = 3600$ * Move decimal two places left -> $36$ * *Step 4: State the final answer clearly.* 4. **Draft the Response (incorporating the steps clearly):** * *Introduction:* State the problem clearly. * *Step 1:* Explain how to convert 15% to a decimal. * *Step 2:* Explain the multiplication step. * *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value). * *Conclusion:* Give the final answer. 5. **Refine the Output (Self-Correction/Polishing during drafting):** * *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15. * *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240. * *Drafting Step 3 (Standard way):* 0.15 × 240 = 36. * *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%. * 10% of 240 = 24 (move the decimal point one place to the left) * 5% is half of 10%, so half of 24 = 12 * Add them together: 24 + 12 = 36. * *Final Answer:* 15% of 240 is 36. 6. **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output). =============== Content ================= Here is the step-by-step solution to find 15% of 240: **Step 1: Convert the percentage to a decimal.** To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left). * 15% = 15 ÷ 100 = **0.15** **Step 2: Multiply the decimal by the number.** Now, multiply the decimal (0.15) by the number you are finding the percentage of (240). * 0.15 × 240 = **36** *(Alternative mental math method for Step 2)*: If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%: * **10% of 240** = 24 (just move the decimal point one place to the left) * **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2) * **Add them together**: 24 + 12 = **36** **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} 15% of 240 is 36. Here is how to calculate it: 1. Convert the percentage to a decimal: 15% = 0.15 2. Multiply the decimal by the number: 0.15 × 240 = 36 ``` #### 4.4.2 Tool Calling GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/GLM-5.1-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Be Tool Call: None Arguments: ijing" Tool Call: None Arguments: } ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: GLM-5.1-FP8 * Tensor Parallelism: 8 * SGLang Version: commit 947927bdb #### 5.1.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5.1-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.78 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4213 Request throughput (req/s): 0.28 Input token throughput (tok/s): 170.54 Output token throughput (tok/s): 117.96 Peak output token throughput (tok/s): 148.00 Peak concurrent requests: 2 Total token throughput (tok/s): 288.50 Concurrency: 1.00 Accept length: 3.48 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3576.31 Median E2E Latency (ms): 2935.97 P90 E2E Latency (ms): 5908.97 P99 E2E Latency (ms): 8588.08 ---------------Time to First Token---------------- Mean TTFT (ms): 290.88 Median TTFT (ms): 282.34 P99 TTFT (ms): 332.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.54 Median TPOT (ms): 6.97 P99 TPOT (ms): 9.04 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.80 Median ITL (ms): 6.81 P95 ITL (ms): 13.51 P99 ITL (ms): 26.99 Max ITL (ms): 29.50 ================================================== ``` #### 5.1.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model zai-org/GLM-5.1-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 411.74 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 499614 Request throughput (req/s): 2.43 Input token throughput (tok/s): 1220.41 Output token throughput (tok/s): 1214.97 Peak output token throughput (tok/s): 2648.00 Peak concurrent requests: 105 Total token throughput (tok/s): 2435.38 Concurrency: 96.30 Accept length: 3.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 39648.76 Median E2E Latency (ms): 39058.12 P90 E2E Latency (ms): 57009.82 P99 E2E Latency (ms): 68880.33 ---------------Time to First Token---------------- Mean TTFT (ms): 20613.80 Median TTFT (ms): 21429.21 P99 TTFT (ms): 29543.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.73 Median TPOT (ms): 36.52 P99 TPOT (ms): 67.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 38.13 Median ITL (ms): 16.57 P95 ITL (ms): 86.01 P99 ITL (ms): 164.88 Max ITL (ms): 1307.02 ================================================== ``` ### 5.2 Accuracy Benchmark The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned. #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.000 Latency: 32.470 s Output throughput: 642.044 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py --port 30000 ``` * Test Result ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.860 subject: anatomy, #q:135, acc: 0.874 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.880 subject: clinical_knowledge, #q:265, acc: 0.932 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.640 subject: college_computer_science, #q:100, acc: 0.900 subject: college_mathematics, #q:100, acc: 0.810 subject: college_medicine, #q:173, acc: 0.873 subject: college_physics, #q:102, acc: 0.912 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.928 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.897 subject: elementary_mathematics, #q:378, acc: 0.937 subject: formal_logic, #q:126, acc: 0.778 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.961 subject: high_school_chemistry, #q:203, acc: 0.847 subject: high_school_computer_science, #q:100, acc: 0.960 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.960 subject: high_school_government_and_politics, #q:193, acc: 0.984 subject: high_school_macroeconomics, #q:390, acc: 0.923 subject: high_school_mathematics, #q:270, acc: 0.696 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.821 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.889 subject: high_school_us_history, #q:204, acc: 0.941 subject: high_school_world_history, #q:237, acc: 0.945 subject: human_aging, #q:223, acc: 0.857 subject: human_sexuality, #q:131, acc: 0.908 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.907 subject: logical_fallacies, #q:163, acc: 0.933 subject: machine_learning, #q:112, acc: 0.830 subject: management, #q:103, acc: 0.942 subject: marketing, #q:234, acc: 0.940 subject: medical_genetics, #q:100, acc: 0.990 subject: miscellaneous, #q:783, acc: 0.959 subject: moral_disputes, #q:346, acc: 0.873 subject: moral_scenarios, #q:895, acc: 0.837 subject: nutrition, #q:306, acc: 0.922 subject: philosophy, #q:311, acc: 0.897 subject: prehistory, #q:324, acc: 0.929 subject: professional_accounting, #q:282, acc: 0.844 subject: professional_law, #q:1534, acc: 0.714 subject: professional_medicine, #q:272, acc: 0.941 subject: professional_psychology, #q:612, acc: 0.913 subject: public_relations, #q:110, acc: 0.791 subject: security_studies, #q:245, acc: 0.878 subject: sociology, #q:201, acc: 0.940 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.596 subject: world_religions, #q:171, acc: 0.936 Total latency: 165.275 Average accuracy: 0.877 ``` ### 5.3 AMD GPU Benchmarks #### 5.3.1 GSM8K Benchmark (MI325/MI35x) * MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends) ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 ``` ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 ``` Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911). # GLM-5.2 Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2 Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points: * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2. Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc` (H200 FP8 cells: `v0.5.14 @ 49e384ce`). Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5; FP8 balanced 1-1-2 = 2; NVFP4 balanced 2-1-3 = 2); high-throughput has no spec. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (\~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within \~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`).
Model Architecture Context
GLM-5.2-FP8 MoE · DSA · 256 experts (top-8) · MTP · FP8 1,048,576
GLM-5.2 MoE · DSA · 256 experts (top-8) · MTP · BF16 1,048,576
GLM-5.2-NVFP4 MoE · DSA · 256 experts (top-8) · MTP · NVFP4 1,048,576
GLM-5.2-MXFP4 MoE · DSA · 256 experts (top-8) · MTP · MXFP4 1,048,576
**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code). **Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4). ## 2. Configuration Tips * **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. * **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute. * **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). * **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. * **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, \~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, \~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points. * **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP. * **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: \~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. * **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, \~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). * **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the same DSA tilelang backends, `--chunked-prefill-size`, and `--watchdog-timeout` as the FP8/BF16 recipes above. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to \~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). * **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP. ## 3. Advanced Usage ### 3.1 Reasoning GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`). **Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: ` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`: | `reasoning_effort` | Injected system line | Effect | | ------------------------------------ | ------------------------ | --------------------------------------------- | | *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** | | `"high"` | `Reasoning Effort: High` | dials reasoning **down** | | `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) | ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What is 15% of 240?"}], extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: 1. **Identify the core question:** The user wants to find 15% of 240. 2. **Convert the percentage to a decimal:** 15% = 0.15 3. **Multiply by the total:** 0.15 * 240 = 36 (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.) Answer: 15% of 240 is **36**. Here is how you can calculate it: 0.15 × 240 = 36 ``` ### 3.2 Tool Calling Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `………` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city. Tool calls: [ { "id": "call_13fcd52146934b7781d06d4a", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"} } ] ``` ### 3.3 HiCache (Hierarchical KV Caching) For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern. ### 3.4 Claude Code Integration GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: ```bash Command theme={null} export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" export API_TIMEOUT_MS="3000000" export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 export CLAUDE_CODE_ATTRIBUTION_HEADER=0 export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" claude ``` Two of these matter specifically for GLM-5.2: * **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse. * **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side. For the full setup (streaming, tool-use, count\_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api). ### 3.5 Context Parallelism Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments: ```bash theme={null} --attn-cp-size 8 \ --enable-prefill-cp \ --cp-strategy interleave \ ``` which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill. When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with ```bash theme={null} --enable-dsa-cache-layer-split \ --attn-cp-size 8 \ --cp-strategy interleave \ ``` With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. # GLM Glyph Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-Glyph ## 1. Model Introduction [Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding. **Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X **Key Features:** * **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving * **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs * **High Performance**: Optimized for both throughput and latency scenarios **Available Models:** * **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph) * **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8) **License:** Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options. ### 3.2 Configuration Tips * **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Thinking Mode Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/Glyph \ --reasoning-parser glm45 \ --tp 4 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="zai-org/Glyph", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. **Disable Thinking Mode:** To disable thinking mode for a specific request: ```python Example theme={null} response = client.chat.completions.create( model="zai-org/Glyph", messages=[{"role": "user", "content": "What is the capital of France?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) ``` #### 4.2.2 Tool Calling Glyph supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path zai-org/Glyph \ --reasoning-parser glm45 \ --tool-call-parser glm45 \ --tp 4 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="zai-org/Glyph", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="zai-org/Glyph", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Model: Glyph * SGLang Version: 0.5.6.post1 **Benchmark Methodology:** We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms. #### 5.1.1 Standard Scenario Benchmark * **Model Deployment** ```bash Command theme={null} python -m sglang.launch_server \ --model zai-org/Glyph \ --tp 2 ``` ##### 5.1.1.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.03 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.59 Input token throughput (tok/s): 358.17 Output token throughput (tok/s): 247.74 Peak output token throughput (tok/s): 251.00 Peak concurrent requests: 3 Total token throughput (tok/s): 605.91 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1702.14 Median E2E Latency (ms): 1361.72 ---------------Time to First Token---------------- Mean TTFT (ms): 22.35 Median TTFT (ms): 22.61 P99 TTFT (ms): 23.76 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.99 Median TPOT (ms): 3.99 P99 TPOT (ms): 4.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 3.99 Median ITL (ms): 3.99 P95 ITL (ms): 4.03 P99 ITL (ms): 4.12 Max ITL (ms): 7.46 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 16.27 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40804 Request throughput (req/s): 4.92 Input token throughput (tok/s): 2438.06 Output token throughput (tok/s): 2507.94 Peak output token throughput (tok/s): 3069.00 Peak concurrent requests: 26 Total token throughput (tok/s): 4946.00 Concurrency: 13.44 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2733.43 Median E2E Latency (ms): 2892.98 ---------------Time to First Token---------------- Mean TTFT (ms): 33.10 Median TTFT (ms): 27.73 P99 TTFT (ms): 49.34 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.33 Median TPOT (ms): 5.39 P99 TPOT (ms): 5.86 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.30 Median ITL (ms): 4.89 P95 ITL (ms): 5.54 P99 ITL (ms): 21.17 Max ITL (ms): 25.14 ================================================== ``` ##### 5.1.1.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 25.67 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252657 Request throughput (req/s): 19.48 Input token throughput (tok/s): 9733.69 Output token throughput (tok/s): 9843.99 Peak output token throughput (tok/s): 13398.00 Peak concurrent requests: 127 Total token throughput (tok/s): 19577.68 Concurrency: 89.49 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4593.75 Median E2E Latency (ms): 4431.03 ---------------Time to First Token---------------- Mean TTFT (ms): 48.66 Median TTFT (ms): 35.88 P99 TTFT (ms): 120.61 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.10 Median TPOT (ms): 9.55 P99 TPOT (ms): 11.00 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.01 Median ITL (ms): 6.51 P95 ITL (ms): 23.19 P99 ITL (ms): 25.54 Max ITL (ms): 52.93 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark ##### 5.1.2.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 201.53 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.05 Input token throughput (tok/s): 30.27 Output token throughput (tok/s): 220.63 Peak output token throughput (tok/s): 251.00 Peak concurrent requests: 2 Total token throughput (tok/s): 250.90 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20151.45 Median E2E Latency (ms): 21576.31 ---------------Time to First Token---------------- Mean TTFT (ms): 2362.23 Median TTFT (ms): 23.03 P99 TTFT (ms): 21310.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.00 Median TPOT (ms): 4.00 P99 TPOT (ms): 4.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.00 Median ITL (ms): 4.00 P95 ITL (ms): 4.05 P99 ITL (ms): 4.08 Max ITL (ms): 5.67 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 118.67 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318306 Total generated tokens (retokenized): 318270 Request throughput (req/s): 0.67 Input token throughput (tok/s): 334.27 Output token throughput (tok/s): 2682.26 Peak output token throughput (tok/s): 3264.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3016.53 Concurrency: 13.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 20387.23 Median E2E Latency (ms): 20466.09 ---------------Time to First Token---------------- Mean TTFT (ms): 132.47 Median TTFT (ms): 27.19 P99 TTFT (ms): 583.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.09 Median TPOT (ms): 5.13 P99 TPOT (ms): 5.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.09 Median ITL (ms): 5.08 P95 ITL (ms): 5.18 P99 ITL (ms): 5.57 Max ITL (ms): 522.26 ================================================== ``` ##### 5.1.2.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 150.00 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1301025 Total generated tokens (retokenized): 1300901 Request throughput (req/s): 2.13 Input token throughput (tok/s): 1059.59 Output token throughput (tok/s): 8673.49 Peak output token throughput (tok/s): 11899.00 Peak concurrent requests: 71 Total token throughput (tok/s): 9733.09 Concurrency: 54.71 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25645.42 Median E2E Latency (ms): 26913.26 ---------------Time to First Token---------------- Mean TTFT (ms): 163.75 Median TTFT (ms): 93.67 P99 TTFT (ms): 426.19 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.27 Median TPOT (ms): 6.39 P99 TPOT (ms): 6.59 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.27 Median ITL (ms): 0.17 P95 ITL (ms): 32.94 P99 ITL (ms): 67.89 Max ITL (ms): 136.00 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark #### 5.1.3.1 Low Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.44 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.57 Input token throughput (tok/s): 2405.19 Output token throughput (tok/s): 242.00 Peak output token throughput (tok/s): 250.00 Peak concurrent requests: 2 Total token throughput (tok/s): 2647.19 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1742.54 Median E2E Latency (ms): 1412.47 ---------------Time to First Token---------------- Mean TTFT (ms): 53.48 Median TTFT (ms): 45.05 P99 TTFT (ms): 98.57 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.01 Median TPOT (ms): 4.01 P99 TPOT (ms): 4.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.01 Median ITL (ms): 4.01 P95 ITL (ms): 4.06 P99 ITL (ms): 4.09 Max ITL (ms): 4.95 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 16.90 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41668 Request throughput (req/s): 4.73 Input token throughput (tok/s): 17753.58 Output token throughput (tok/s): 2465.75 Peak output token throughput (tok/s): 3005.00 Peak concurrent requests: 25 Total token throughput (tok/s): 20219.33 Concurrency: 13.68 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2890.33 Median E2E Latency (ms): 3069.55 ---------------Time to First Token---------------- Mean TTFT (ms): 41.46 Median TTFT (ms): 31.75 P99 TTFT (ms): 93.18 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.52 Median TPOT (ms): 5.58 P99 TPOT (ms): 6.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.48 Median ITL (ms): 5.13 P95 ITL (ms): 5.93 P99 ITL (ms): 20.76 Max ITL (ms): 36.01 ================================================== ``` ##### 5.1.3.3 High Concurrency * **Benchmark Command**: ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model zai-org/Glyph \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 35.54 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169994 Request throughput (req/s): 9.01 Input token throughput (tok/s): 35848.57 Output token throughput (tok/s): 4783.96 Peak output token throughput (tok/s): 8396.00 Peak concurrent requests: 80 Total token throughput (tok/s): 40632.53 Concurrency: 59.26 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6580.96 Median E2E Latency (ms): 6248.74 ---------------Time to First Token---------------- Mean TTFT (ms): 345.27 Median TTFT (ms): 96.06 P99 TTFT (ms): 2823.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.26 Median TPOT (ms): 12.53 P99 TPOT (ms): 23.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.76 Median ITL (ms): 6.57 P95 ITL (ms): 27.66 P99 ITL (ms): 91.24 Max ITL (ms): 2609.64 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-questions 200 ``` * Test Result ```text Output theme={null} Accuracy: 0.890 Invalid: 0.000 Latency: 3.718 s Output throughput: 5245.606 token/s ``` # GLM-OCR Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-OCR ## 1. Model Introduction [GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization. The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts. **Hardware Support:** NVIDIA B200/H100/H200 **Key Features:** * **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction. * **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts. * **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments. * **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems. For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding. ### 3.2 Configuration Tips * **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT. * **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency. * **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 OCR Image Processing GLM-OCR supports OCR tasks on various document types. Here's a basic example: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Please extract all text from this image." } ] } ] start = time.time() response = client.chat.completions.create( model="zai-org/GLM-OCR", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 2.29s Generated text: CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` #### 4.2.2 Complex Document Processing GLM-OCR excels at processing complex documents including: * **Tables**: Accurate extraction of tabular data with structure preservation * **Formulas**: Mathematical formula recognition * **Code Documents**: Source code extraction from screenshots * **Seals and Stamps**: Recognition of seals and stamps in documents * **Multi-layout Documents**: Mixed content with text, images, and tables ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) # Example: Processing a document with tables messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "YOUR_DOCUMENT_IMAGE_URL" } }, { "type": "text", "text": "Please extract the table content from this document and format it as markdown." } ] } ] response = client.chat.completions.create( model="zai-org/GLM-OCR", messages=messages, max_tokens=4096 ) print(response.choices[0].message.content) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.1.1 OCRBench Benchmark * Benchmark Command ```bash Command theme={null} python3 -m lmms_eval \ --model openai_compatible \ --model_args "model_version=zai-org/GLM-OCR" \ --tasks ocrbench \ --batch_size 128 \ --log_samples \ --log_samples_suffix "openai_compatible" \ --output_path ./logs ``` * Test Result
Tasks Version Filter n-shot Metric Value Stderr
ocrbench Yaml none 0 ocrbench\_accuracy ↑ 0.806 N/A
#### 5.1.2 OmniDocBench V1.5 GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks. # DiffusionGemma Source: https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma ## 1. Model Introduction DiffusionGemma is a uniform-state (renoising) block-diffusion language model from Google. An encoder builds causal context, and a decoder denoises a fixed-length bidirectional canvas of `canvas_length` tokens. The `Gemma4Renoise` sampler runs `max_denoising_steps` reverse steps over the canvas, feeding the previous step's logits back as self-conditioning and emitting the greedy argmax of the processed logits. **Key Features:** * **Uniform-State Renoising**: The canvas starts from random tokens and is refined each step by accepting confident positions and re-noising the rest, with no mask token. * **Encoder / Decoder Canvas**: The encoder produces causal context KV, the decoder attends bidirectionally over the canvas. * **Self-Conditioning**: Each step conditions on the previous step's logits. * **EntropyBound Acceptance**: Each step accepts the lowest-entropy canvas positions within an entropy budget and re-noises the rest. * **StableAndConfident Stopping**: A canvas stops early once it is stable and confident. * **MoE Architecture**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference. * **Multimodal Input**: Accepts text and image inputs (via a \~550M vision encoder) and generates text output. **Available Models:**
Model Architecture Parameters
[google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it) MoE, uniform-state diffusion (text + image) 25.2B total / 3.8B active
**Architecture Specifications:** | Spec | Value | | -------------------- | ------------------------------- | | Total Parameters | 25.2B | | Active Parameters | 3.8B | | Layers | 30 | | Sliding Window | 1024 tokens | | Context Length | Up to 256K tokens | | Canvas Length | 256 | | Vocabulary Size | 262K | | Experts | 8 active / 128 total + 1 shared | | Supported Modalities | Text, Image | | Vision Encoder | \~550M parameters | **License:** Refer to the model card for license details. ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The checkpoint ships its own modeling code, so `--trust-remote-code` is required when serving. ## 3. Model Deployment ### 3.1 Basic Configuration The required runtime settings are applied automatically for `Gemma4Renoise` (the Triton attention backend, eager mode, and unchunked prefill, needed because the full-attention head\_dim is 512 and the canvas uses bidirectional attention), so a default launch works: ```bash Command theme={null} sglang serve \ --model-path google/diffusiongemma-26B-A4B-it \ --dllm-algorithm Gemma4Renoise \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### 3.2 Configuration Tips **dLLM-Specific Parameters:**
Parameter Description Recommended Value
`--dllm-algorithm` Diffusion decoding algorithm `Gemma4Renoise`
`--trust-remote-code` Required to load the checkpoint's modeling code Always enabled
`--dllm-algorithm-config` Optional YAML overriding the renoise schedule Checkpoint defaults
The attention backend, eager mode, and unchunked prefill are selected automatically for `Gemma4Renoise`, so they do not need to be passed on the command line. Sampling is governed by the renoise schedule. Request-level `logprobs`, penalties, `logit_bias`, and grammar / structured output (`json_schema` / `regex` / `ebnf` / `structural_tag`) are not applied and are rejected with a 400. Core sampling controls (`temperature`, `top_k`, `top_p`) are accepted but have no effect. Streaming is block-level: one fully-denoised canvas per chunk. **Gemma4Renoise Config** (defaults follow the checkpoint's `generation_config.json`): ```yaml Config theme={null} # Number of reverse denoising steps per canvas. max_denoising_steps: 48 # Optional. Makes the renoise sampling reproducible (also shared across TP ranks). seed: 1234 sampler_config: # Entropy budget. Accept the lowest-entropy canvas positions within this bound each step (the rest are re-noised). entropy_bound: 0.1 # Linear temperature schedule applied over the denoising steps. temperature_schedule: t_min: 0.4 t_max: 0.8 # Stop early once the canvas is stable and confident. stopping_config: confidence_threshold: 0.005 stability_threshold: 1 ``` ## 4. Model Invocation ### 4.1 Deployment Start the server with the command from [Section 3.1](#3-1-basic-configuration). ### 4.2 Basic Usage ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[ {"role": "user", "content": "What are the key differences between TCP and UDP?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` ### 4.3 Streaming Streaming emits one fully-denoised canvas per chunk. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/diffusiongemma-26B-A4B-it", messages=[ {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` ## 5. Benchmark ### 5.1 Speed Benchmark Not benchmarked for speed. ### 5.2 Accuracy Benchmark Full test splits, every item scored (no failed-request exclusions). Text MCQ benchmarks use greedy generate-and-parse, MATH uses boxed-answer extraction plus sympy equivalence. MMLU, ARC-Challenge, and MATH-500 are the mean of two independent server launches.
Benchmark Score
GSM8K 95.4%
ARC-Challenge 91.6%
HumanEval 92.7% pass\@1
MMLU 76.2%
MMLU-Pro 73.7%
GSM-Symbolic 92.2%
MATH-500 72.1%
AIME-2026 10.0%
HMMT-Feb-2025 10.0%
GPQA-main 59.2%
Multimodal, full standard split per task (MMMU / MMMU-Pro / MMStar / AI2D as multiple-choice, MathVista testmini, DocVQA by ANLS, ChartQA by relaxed accuracy):
Multimodal benchmark Score
MMMU (val, MC) 64.9%
MMMU-Pro (standard 10-opt, MC) 57.3%
MathVista (testmini) 68.4%
DocVQA (val) 85.9%
ChartQA (test) 61.7%
AI2D (test) 78.7%
MMStar (val) 65.9%
# EmbeddingGemma Source: https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma Serve Google's EmbeddingGemma text embedding model with SGLang. ## Overview [EmbeddingGemma](https://huggingface.co/google/embeddinggemma-300m) is Google's 300M-parameter text embedding model. SGLang detects its bidirectional Gemma 3 encoder, applies normalized mean pooling, and serves embeddings through the OpenAI-compatible `/v1/embeddings` endpoint. On NVIDIA CUDA, SGLang uses breakable CUDA graph (BCG) for its complete prefill by default. It also disables prefix caching and chunked prefill, which are incompatible with this bidirectional encoder. ## Prerequisites * NVIDIA CUDA GPU. * A Hugging Face account that has accepted the [EmbeddingGemma license](https://huggingface.co/google/embeddinggemma-300m). * A Hugging Face access token. Export it before starting the server so it can download the gated checkpoint: ```bash theme={null} export HF_TOKEN= ``` Install an SGLang build that includes EmbeddingGemma support: ```bash theme={null} pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` ## Start the server The standard configuration detects EmbeddingGemma and enables embedding mode, BCG, and the checkpoint's BF16 dtype automatically: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-300m \ --host 0.0.0.0 ``` ### Hopper performance defaults On H100 and H200, SGLang automatically selects FA3 and captures BCG through 16,384 tokens, covering eight 2K embedding requests in one replay. No extra performance flags are required for this workload. To capture larger aggregate prefills, raise the BCG tier explicitly: ```bash theme={null} sglang serve \ --model-path google/embeddinggemma-300m \ --cuda-graph-max-bs-prefill 32768 \ --host 0.0.0.0 ``` EmbeddingGemma automatically enables batch tokenization for list-valued embedding requests, so do not add a separate tokenizer batching flag. ## Create embeddings Send one string or a batch of strings to the OpenAI-compatible endpoint: ```bash theme={null} curl http://127.0.0.1:30000/v1/embeddings \ -H 'Content-Type: application/json' \ -d '{ "model": "google/embeddinggemma-300m", "input": [ "A short guide to serving text embeddings.", "Vector search retrieves semantically similar documents." ], "encoding_format": "float" }' ``` See [OpenAI-compatible embedding APIs](/docs/basic_usage/openai_api_embeddings) for Python and OpenAI client examples. ## Deployment behavior EmbeddingGemma performs bidirectional attention over the complete input, so reusing a prefix KV cache or splitting the input into chunked prefills would produce incorrect attention states. SGLang applies the required settings automatically: * disables RadixAttention prefix caching; * disables chunked prefill; * disables the decode CUDA graph because this is an embedding-only model; * uses BCG for CUDA prefill; * uses the FlashAttention raw-K/V path when the prefill backend is FA3 or FA4 on supported Hopper and Blackwell CUDA GPUs. No prefill CUDA-graph override is required for this recipe. Keep BCG enabled to use the optimized EmbeddingGemma path. # Gemma 4 Source: https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4 ## 1. Model Introduction Gemma 4 is Google's next-generation family of open models, building on the Gemma 3 architecture with improved performance, MoE variants, and multimodal support for text, vision, and audio. **Key Features:** * **Hybrid Attention**: Combines sliding window and full attention layers for efficient long-context processing * **Multimodal**: Supports text, image, and audio inputs via dedicated vision and audio encoders * **MoE Variant**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference * **Per-Layer Embeddings (PLE)**: Layer-specific token embeddings for enhanced representations * **Reasoning**: Built-in thinking mode with `gemma4` reasoning parser * **Tool Calling**: Function call support with streaming via `gemma4` tool call parser * **Fused Operations**: Triton-optimized RMSNorm + residual + scalar kernels **Available Models:**
Model Architecture Parameters
[google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) Dense \~2B
[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) Dense \~4B
[google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) Dense 12B
[google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) Dense 31B
[google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) MoE 26B total / 4B active
## 2. SGLang Installation Gemma 4 (including the encoder-free unified 12B, [sgl-project/sglang#27167](https://github.com/sgl-project/sglang/pull/27167)) is supported on SGLang main. Install it together with the matching transformers commit: ```bash Command theme={null} # Install SGLang from main pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Install transformers with Gemma 4 support (encoder-free unified family included) pip install 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897' ``` ### Docker `lmsysorg/sglang:latest` (CUDA 13.0, multi-arch `amd64` + `arm64`) runs on both Hopper (H200) and Blackwell (B200 / GB200 / GB300): ```bash Command theme={null} docker run --gpus all --ipc=host --shm-size 32g \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ lmsysorg/sglang:latest \ sglang serve --model-path google/gemma-4-12B-it \ --reasoning-parser gemma4 --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant. ### 3.2 Configuration Tips * SGLang automatically selects the Triton attention backend for Gemma 4 models (required for bidirectional image-token attention during prefill). * **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for text but applies *causal* attention to image tokens. For multimodal (image) workloads on B200, pass `--attention-backend triton` to restore bidirectional image-token attention and full vision quality. Text-only and audio workloads are unaffected by the default. * **Gemma 4 26B-A4B on B200**: Use `--mem-fraction-static 0.75` to leave workspace headroom for the Triton MoE path. * For the 26B-A4B MoE model, consider `--tp 2` for high-throughput workloads. * **Speculative Decoding (MTP)**: Each Gemma 4 variant ships with a paired `*-assistant` draft model that enables NEXTN multi-token prediction. Enable it via the selector above, or pass `--speculative-algorithm NEXTN --speculative-draft-model-path google/gemma-4--it-assistant --speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-eagle-topk 1`. MTP can significantly reduce latency for interactive use cases. The 26B-A4B MoE model requires `--tp 2` when MTP is enabled. * **QAT checkpoints**: Toggle **Checkpoint → QAT** in the selector to target the `qat-q4_0-unquantized` releases. These keep bf16 weights, so memory and TP requirements match the standard checkpoints, and each has a matching `*-qat-q4_0-unquantized-assistant` draft model for MTP. * Hardware requirements:
Model Hardware TP
gemma-4-E2B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
gemma-4-E4B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
gemma-4-12B-it 1x H200 / 1x B200 / 1x B300 1
gemma-4-31B-it 2x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 2 (H200) / 1 (B200/B300/AMD)
gemma-4-26B-A4B-it 1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X 1
### 3.3 AMD GPU Deployment (MI300X / MI325X / MI355X) SGLang automatically selects the correct attention backend on AMD GPUs. For the small E-models (`gemma-4-E2B-it`, `gemma-4-E4B-it`), disable AITER on AMD GPUs and use the same command line otherwise: ```bash Command theme={null} SGLANG_USE_AITER=0 sglang serve --model-path google/gemma-4-E4B-it \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` For `gemma-4-31B-it` and `gemma-4-26B-A4B-it`, the same commands above work on MI300X, MI325X, and MI355X without additional command-line changes. > **Status**: AMD benchmarks are available in [Section 5.1](#5-1-speed-benchmark). ## 4. Model Invocation Deploy gemma-4-26B-A4B-it (MoE) with all features enabled: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it \ --reasoning-parser gemma4 \ --tool-call-parser gemma4 \ --host 0.0.0.0 --port 30000 ``` #### Speculative Decoding (MTP) Server Commands Each Gemma 4 variant ships with a paired `*-assistant` draft model for NEXTN multi-token prediction. Use the commands below to enable MTP for the corresponding target model. These match the configuration generated when you toggle **Speculative Decoding (MTP) → Enabled** in the [interactive selector](#3-1-basic-configuration). ```bash Command theme={null} # Gemma 4 E2B + MTP sglang serve \ --model-path google/gemma-4-E2B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-E2B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 E4B + MTP sglang serve \ --model-path google/gemma-4-E4B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-E4B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 12B + MTP (~35% faster single-stream decode on H200) sglang serve \ --model-path google/gemma-4-12B-it \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-12B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 31B + MTP sglang serve \ --model-path google/gemma-4-31B-it \ --tp-size 2 \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-31B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ```bash Command theme={null} # Gemma 4 26B-A4B + MTP sglang serve \ --model-path google/gemma-4-26B-A4B-it \ --tp-size 2 \ --speculative-algorithm NEXTN \ --speculative-draft-model-path google/gemma-4-26B-A4B-it-assistant \ --speculative-num-steps 5 \ --speculative-num-draft-tokens 6 \ --speculative-eagle-topk 1 \ --mem-fraction-static 0.85 ``` ### 4.1 Basic Usage ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "What are the key differences between TCP and UDP?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} The fundamental difference between **TCP (Transmission Control Protocol)** and **UDP (User Datagram Protocol)** lies in how they prioritize data integrity versus speed. ### 1. Connection Type * **TCP (Connection-Oriented):** Before any data is sent, TCP performs a "three-way handshake." The sender and receiver exchange signals to establish a formal connection. * **UDP (Connectionless):** UDP does not establish a connection. It simply starts blasting packets to the destination IP address without checking if the receiver is ready. ### 2. Reliability and Error Checking * **TCP (Reliable):** If a packet is lost or arrives corrupted, TCP detects the error and retransmits the missing data. * **UDP (Unreliable):** If a packet is lost or corrupted, it is simply discarded. There is no mechanism to ask for a retransmission. ### 3. Ordering of Data * **TCP (Ordered):** Segments are assigned sequence numbers and reassembled in the correct order. * **UDP (Unordered):** Packets may arrive in a different order than sent. ### 4. Speed and Overhead * **TCP (Slower):** Managing connections, tracking, and retransmissions adds significant overhead. * **UDP (Faster):** No handshake, no tracking — extremely fast and ideal for real-time needs. | Feature | TCP | UDP | | :--- | :--- | :--- | | **Connection** | Connection-oriented | Connectionless | | **Reliability** | Guaranteed delivery | Best-effort | | **Ordering** | Maintains strict order | No guaranteed order | | **Speed** | Slower (High overhead) | Faster (Low overhead) | ```
### 4.2 Vision Input Gemma 4 multimodal variants accept images alongside text: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://farm4.staticflickr.com/3175/2653711032_804ff86d81_z.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=1024 ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} A vertical, full shot shows a girl and a boy standing in front of a giant teddy bear. The boy, who is on the left, is of South Asian descent, has short dark hair, and is smiling at the camera. He is wearing a navy blue sweatshirt with a white collar, blue jeans, and white, black, and red sneakers. The girl, on the right, is also of South Asian descent and has long, dark hair. She is smiling at the camera and is wearing a pink t-shirt, a white long-sleeve shirt underneath, blue jeans, and pink sneakers. The giant teddy bear is light brown and is standing behind the two children. The bear has large, dark eyes and a black nose. In the background, on the left, there is a large wooden basket filled with small teddy bears. To the left of the basket, an American flag is hanging on the wall. On the right side of the image, there is a green leafy plant. The floor is a dark purple carpet. The lighting is bright and even. ```
### 4.3 Reasoning (Thinking Mode) Gemma 4 supports hybrid reasoning. Thinking is **not enabled by default** — pass `chat_template_kwargs: {"enable_thinking": true}` via `extra_body` to activate it. The reasoning parser separates thinking and content, returning the thinking process via `reasoning_content` in the streaming response. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "Solve step by step: If a train travels at 60 km/h for 2.5 hours, how far does it go?"} ], max_tokens=4096, stream=True, extra_body={"chat_template_kwargs": {"enable_thinking": True}} ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ```
Example Output ```text Output theme={null} =============== Thinking ================= * Input: Speed = 60 km/h, Time = 2.5 hours. * Goal: Find the distance traveled. * Distance = Speed × Time. * Step 1: Identify given values. Speed = 60 km/h, Time = 2.5 hours * Step 2: Formula. Distance = Speed × Time * Step 3: Calculation. 60 × 2.5 Mental math: 60 × 2 = 120; 60 × 0.5 = 30; 120 + 30 = 150. * Step 4: Final Result. 150 km. =============== Content ================= To find the distance traveled, you can follow these steps: ### 1. Identify the given information: * **Speed:** 60 km/h * **Time:** 2.5 hours ### 2. Use the distance formula: Distance = Speed × Time ### 3. Substitute the values: Distance = 60 km/h × 2.5 hours ### 4. Perform the calculation: * 60 × 2 = 120 * 60 × 0.5 = 30 * 120 + 30 = 150 **Final Answer: The train travels 150 km.** ```
### 4.4 Tool Calling Gemma 4 supports function calling with the `gemma4` tool call parser. Enable it during deployment with `--tool-call-parser gemma4`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="google/gemma-4-26B-A4B-it", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"} ], tools=tools, stream=True ) thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if hasattr(delta, 'tool_calls') and delta.tool_calls: if has_thinking and thinking_started: print("\n=============== Tool Calls ================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") if delta.content: print(delta.content, end="", flush=True) print() ```
Example Output ```text Output theme={null} =============== Tool Calls ================ Tool Call: get_weather Arguments: {"location": "Tokyo"} ```
### 4.5 Audio Input The audio-capable Gemma 4 variants (`gemma-4-E2B-it`, `gemma-4-E4B-it`, `gemma-4-12B-it`) accept raw audio alongside text. Pass the waveform as a base64 `audio_url` data URI (16 kHz mono WAV works well), or as OpenAI's `input_audio` part with the base64 bytes in `data` and a `format` of `wav`: ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("sample.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() response = client.chat.completions.create( model="google/gemma-4-12B-it", messages=[ { "role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Transcribe the speech in this audio exactly."}, ], } ], max_tokens=256, temperature=0, ) print(response.choices[0].message.content) ```
Example Output ```text Output theme={null} Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. ```
For best ASR quality, use the recommended transcription prompt structure: ```text Prompt theme={null} Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text. Follow these specific instructions for formatting the answer: * Only output the transcription, with no newlines. * When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three. ``` For speech translation (AST), ask for the transcription in the source language first, then the translation: *"Transcribe the following speech segment in , then translate it into . ..."* ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 * SGLang Version: gemma4 branch #### gemma-4-E2B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-E2B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 17.44 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.57 Output token throughput (tok/s): 242.03 Total token throughput (tok/s): 591.94 Mean TTFT (ms): 50.19 Median TTFT (ms): 54.22 Mean TPOT (ms): 3.99 Median ITL (ms): 4.05 ================================================== ``` **Latency Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.05 Total input tokens: 6097 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.55 Output token throughput (tok/s): 233.84 Total token throughput (tok/s): 571.69 Mean TTFT (ms): 109.59 Median TTFT (ms): 112.62 Mean TPOT (ms): 4.01 Median ITL (ms): 4.04 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 51.73 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 19.33 Output token throughput (tok/s): 9876.36 Peak output token throughput (tok/s): 13863.00 Total token throughput (tok/s): 19791.14 Mean TTFT (ms): 86.57 Mean TPOT (ms): 9.56 Median ITL (ms): 5.99 ================================================== ``` **Throughput Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 89.07 Total input tokens: 617799 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 11.23 Output token throughput (tok/s): 5735.75 Peak output token throughput (tok/s): 12823.00 Total token throughput (tok/s): 12672.23 Mean TTFT (ms): 636.46 Mean TPOT (ms): 16.34 Median ITL (ms): 5.68 ================================================== ``` #### gemma-4-E4B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-E4B-it ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 24.49 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.41 Output token throughput (tok/s): 172.32 Total token throughput (tok/s): 421.45 Mean TTFT (ms): 52.76 Median TTFT (ms): 53.66 Mean TPOT (ms): 5.64 Median ITL (ms): 5.74 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.04 Total input tokens: 6124 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 168.54 Total token throughput (tok/s): 413.13 Mean TTFT (ms): 110.15 Median TTFT (ms): 108.24 Mean TPOT (ms): 5.66 Median ITL (ms): 5.73 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 72.95 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 13.71 Output token throughput (tok/s): 7002.68 Peak output token throughput (tok/s): 9878.00 Total token throughput (tok/s): 14032.60 Mean TTFT (ms): 166.33 Mean TPOT (ms): 13.36 Median ITL (ms): 8.88 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 108.99 Total input tokens: 616952 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 9.18 Output token throughput (tok/s): 4687.38 Peak output token throughput (tok/s): 9277.00 Total token throughput (tok/s): 10348.25 Mean TTFT (ms): 626.17 Mean TPOT (ms): 20.00 Median ITL (ms): 8.64 ================================================== ``` #### gemma-4-31B-it (2x H200, TP=2) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-31B-it --tp 2 ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.05 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.19 Output token throughput (tok/s): 79.55 Total token throughput (tok/s): 194.55 Mean TTFT (ms): 72.77 Median TTFT (ms): 75.05 Mean TPOT (ms): 12.32 Median ITL (ms): 12.53 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 53.78 Total input tokens: 6162 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.19 Output token throughput (tok/s): 78.46 Total token throughput (tok/s): 193.03 Mean TTFT (ms): 143.35 Median TTFT (ms): 146.85 Mean TPOT (ms): 12.37 Median ITL (ms): 12.48 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 182.00 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 5.49 Output token throughput (tok/s): 2806.82 Peak output token throughput (tok/s): 3798.00 Total token throughput (tok/s): 5624.56 Mean TTFT (ms): 324.67 Mean TPOT (ms): 33.95 Median ITL (ms): 25.44 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 236.46 Total input tokens: 621630 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 4.23 Output token throughput (tok/s): 2160.42 Peak output token throughput (tok/s): 3745.00 Total token throughput (tok/s): 4789.30 Mean TTFT (ms): 952.02 Mean TPOT (ms): 44.17 Median ITL (ms): 26.81 ================================================== ``` #### gemma-4-26B-A4B-it (MoE, 1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it ``` > **Tip**: Consider `--tp 2` for high-throughput workloads. **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.00 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 168.81 Total token throughput (tok/s): 412.85 Mean TTFT (ms): 103.74 Median TTFT (ms): 46.57 Mean TPOT (ms): 5.60 Median ITL (ms): 5.78 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.31 Total input tokens: 6164 Total input vision tokens: 5340 Total generated tokens: 4220 Request throughput (req/s): 0.40 Output token throughput (tok/s): 166.70 Total token throughput (tok/s): 410.20 Mean TTFT (ms): 129.22 Median TTFT (ms): 132.54 Mean TPOT (ms): 5.68 Median ITL (ms): 5.75 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 138.98 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 7.20 Output token throughput (tok/s): 3675.81 Peak output token throughput (tok/s): 4799.00 Total token throughput (tok/s): 7365.91 Mean TTFT (ms): 153.77 Mean TPOT (ms): 25.95 Median ITL (ms): 20.23 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 186.38 Total input tokens: 621146 Total input vision tokens: 534000 Total generated tokens: 510855 Request throughput (req/s): 5.37 Output token throughput (tok/s): 2740.86 Peak output token throughput (tok/s): 4962.00 Total token throughput (tok/s): 6073.47 Mean TTFT (ms): 854.71 Mean TPOT (ms): 34.64 Median ITL (ms): 19.08 ================================================== ``` #### gemma-4-31B-it (1x MI300X, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-31B-it ``` > **Note**: The 31B dense model fits on a single MI300X (192 GB VRAM) at TP=1, unlike H200 (141 GB) which requires TP=2. **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 103.55 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.10 Output token throughput (tok/s): 40.75 Total token throughput (tok/s): 99.67 Mean TTFT (ms): 152.35 Median TTFT (ms): 169.66 Mean TPOT (ms): 24.13 Median ITL (ms): 24.23 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 441.59 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 2.26 Output token throughput (tok/s): 1156.85 Peak output token throughput (tok/s): 1759.00 Total token throughput (tok/s): 2318.19 Mean TTFT (ms): 819.22 Mean TPOT (ms): 82.51 Median ITL (ms): 63.45 ================================================== ``` #### gemma-4-26B-A4B-it (MoE, 1x MI300X, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-26B-A4B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.73 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.23 Output token throughput (tok/s): 96.49 Total token throughput (tok/s): 236.00 Mean TTFT (ms): 185.58 Median TTFT (ms): 90.18 Mean TPOT (ms): 9.78 Median ITL (ms): 9.57 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 219.43 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 4.56 Output token throughput (tok/s): 2328.05 Peak output token throughput (tok/s): 3500.00 Total token throughput (tok/s): 4665.16 Mean TTFT (ms): 168.44 Mean TPOT (ms): 41.23 Median ITL (ms): 29.31 ================================================== ``` #### gemma-4-12B-it (1x H200, TP=1) Server Launch Command: ```bash Command theme={null} sglang serve --model-path google/gemma-4-12B-it ``` **Latency Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 38.66 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.26 Output token throughput (tok/s): 109.15 Total token throughput (tok/s): 266.94 Mean TTFT (ms): 33.08 Median TTFT (ms): 33.71 Mean TPOT (ms): 9.02 Median ITL (ms): 9.19 ================================================== ``` **Latency Benchmark (Image)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang-oai-chat \ --host 0.0.0.0 --port 30000 \ --dataset-name image --image-count 2 --image-resolution 720p \ --random-input-len 128 --random-output-len 1024 \ --num-prompts 10 --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.36 Total input vision tokens: 5320 Total generated tokens: 4220 Request throughput (req/s): 0.25 Output token throughput (tok/s): 107.23 Total token throughput (tok/s): 263.62 Mean TTFT (ms): 94.98 Median TTFT (ms): 97.33 Mean TPOT (ms): 9.08 Median ITL (ms): 9.17 ================================================== ``` **Throughput Benchmark (Text)** ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 130.44 Total input tokens: 512842 Total generated tokens: 510855 Request throughput (req/s): 7.67 Output token throughput (tok/s): 3916.46 Total token throughput (tok/s): 7848.15 Mean TTFT (ms): 207.49 Median TTFT (ms): 76.95 Mean TPOT (ms): 24.38 Median ITL (ms): 17.89 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 147.57 Total input tokens: 619609 Total input vision tokens: 532000 Total generated tokens: 510855 Request throughput (req/s): 6.78 Output token throughput (tok/s): 3461.79 Total token throughput (tok/s): 7660.54 Mean TTFT (ms): 438.40 Median TTFT (ms): 129.83 Mean TPOT (ms): 27.12 Median ITL (ms): 19.16 ================================================== ``` #### gemma-4-12B-it (1x B200, TP=1) Server Launch Command: ```bash Command theme={null} # Text/audio: the sm100 default (trtllm_mha) is fastest. # For image workloads add --attention-backend triton (bidirectional image attention). sglang serve --model-path google/gemma-4-12B-it --attention-backend triton ``` **Latency Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 30.46 Output token throughput (tok/s): 138.55 Total token throughput (tok/s): 338.85 Mean TTFT (ms): 28.14 Median TTFT (ms): 29.74 Mean TPOT (ms): 7.08 Median ITL (ms): 7.26 ================================================== ``` **Latency Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 31.43 Total input vision tokens: 5320 Total generated tokens: 4220 Request throughput (req/s): 0.32 Output token throughput (tok/s): 134.26 Total token throughput (tok/s): 329.57 Mean TTFT (ms): 115.51 Median TTFT (ms): 74.27 Mean TPOT (ms): 7.14 Median ITL (ms): 7.24 ================================================== ``` **Throughput Benchmark (Text)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 92.94 Request throughput (req/s): 10.76 Output token throughput (tok/s): 5496.55 Total token throughput (tok/s): 11014.49 Mean TTFT (ms): 120.89 Median TTFT (ms): 45.00 Mean TPOT (ms): 17.23 Median ITL (ms): 14.30 ================================================== ``` **Throughput Benchmark (Image)** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Max request concurrency: 100 Successful requests: 998 Benchmark duration (s): 107.82 Total input tokens: 617971 Total input vision tokens: 530936 Total generated tokens: 508951 Request throughput (req/s): 9.26 Output token throughput (tok/s): 4720.29 Total token throughput (tok/s): 10451.68 Mean TTFT (ms): 425.89 Median TTFT (ms): 109.57 Mean TPOT (ms): 19.45 Median ITL (ms): 15.11 ================================================== ``` > **Performance tuning**: On B200, raising `--scheduler-recv-interval` to 16 lifted text throughput from 5497 to 5673 tok/s output (≈ +3%) at concurrency 100 with no accuracy change, by reducing the scheduler's per-step Python overhead. It is a safe, low-risk knob for high-concurrency serving. ### 5.2 Accuracy Benchmark **Test Environment:** * Hardware: H200 * SGLang Version: gemma4 branch #### MMLU
Model Humanities Social Sciences STEM Other Overall
gemma-4-E2B-it 0.621 0.739 0.830 0.736 **0.720**
gemma-4-E4B-it 0.703 0.862 0.902 0.825 **0.810**
gemma-4-12B-it 0.784 0.888 0.946 0.861 **0.859**
gemma-4-31B-it 0.878 0.921 0.884 0.911 **0.896**
gemma-4-26B-A4B-it 0.853 0.906 0.938 0.886 **0.891**
#### GSM8K
Model Accuracy Invalid Latency (s) Output Throughput (tok/s)
gemma-4-E2B-it 0.170 0.000 3.990 8041.739
gemma-4-E4B-it 0.745 0.000 4.174 4672.030
gemma-4-12B-it 0.431 0.052 55.105 6580.229
gemma-4-31B-it 0.805 0.005 16.148 1559.914
gemma-4-26B-A4B-it 0.450 0.010 13.001 4089.457
> **Note**: These GSM8K numbers use the raw few-shot completion harness (`sglang.test.few_shot_gsm8k`). `gemma-4-12B-it` is reasoning-oriented and is under-elicited by raw few-shot prompting; with the chat template it scores **0.950** on the same 1319 GSM8K test questions (`sglang.test.run_eval --eval-name gsm8k`). #### gemma-4-12B-it with sgl-eval `gemma-4-12B-it` is reasoning-oriented and answers verbosely (step-by-step) rather than emitting a terse final line. Strict last-line `Answer: $LETTER` extraction (as in `sglang.test.run_eval`) therefore undercounts its correct answers. [sgl-eval](https://github.com/sgl-project/sgl-eval) — sgl-project's evaluation CLI, which uses robust answer extraction — gives a faithful score on the served model: | Benchmark | Examples | Accuracy | | --------- | -------- | --------- | | MMLU | 2000 | **0.878** | | GSM8K | 1319 | **0.960** | Reproduce against a running server (`--base-url` points at your endpoint): ```bash Command theme={null} pip install git+https://github.com/sgl-project/sgl-eval # Sanity-check the endpoint sgl-eval ping --base-url http://localhost:30000/v1 # Run the benchmarks (greedy, single-shot) sgl-eval run gsm8k --base-url http://localhost:30000/v1 sgl-eval run mmlu --base-url http://localhost:30000/v1 --num-examples 2000 ``` #### MMMU
Model Overall
gemma-4-E2B-it **0.307**
gemma-4-E4B-it **0.396**
gemma-4-12B-it **0.683**
gemma-4-31B-it **0.589**
gemma-4-26B-A4B-it **0.549**
MMMU detailed scores (per domain) **gemma-4-E2B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.45}, "Art": {"num": 30, "acc": 0.5}, "Art_Theory": {"num": 30, "acc": 0.467}, "Design": {"num": 30, "acc": 0.5}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.26}, "Accounting": {"num": 30, "acc": 0.367}, "Economics": {"num": 30, "acc": 0.233}, "Finance": {"num": 30, "acc": 0.2}, "Manage": {"num": 30, "acc": 0.233}, "Marketing": {"num": 30, "acc": 0.267}, "Overall-Science": {"num": 150, "acc": 0.273}, "Biology": {"num": 30, "acc": 0.233}, "Chemistry": {"num": 30, "acc": 0.267}, "Geography": {"num": 30, "acc": 0.367}, "Math": {"num": 30, "acc": 0.233}, "Physics": {"num": 30, "acc": 0.267}, "Overall-Health and Medicine": {"num": 150, "acc": 0.273}, "Basic_Medical_Science": {"num": 30, "acc": 0.5}, "Clinical_Medicine": {"num": 30, "acc": 0.233}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.233}, "Pharmacy": {"num": 30, "acc": 0.3}, "Public_Health": {"num": 30, "acc": 0.1}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.4}, "History": {"num": 30, "acc": 0.4}, "Literature": {"num": 30, "acc": 0.567}, "Sociology": {"num": 30, "acc": 0.333}, "Psychology": {"num": 30, "acc": 0.3}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.252}, "Agriculture": {"num": 30, "acc": 0.333}, "Architecture_and_Engineering": {"num": 30, "acc": 0.267}, "Computer_Science": {"num": 30, "acc": 0.233}, "Electronics": {"num": 30, "acc": 0.1}, "Energy_and_Power": {"num": 30, "acc": 0.3}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.307}} ``` **gemma-4-E4B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.458}, "Art": {"num": 30, "acc": 0.433}, "Art_Theory": {"num": 30, "acc": 0.567}, "Design": {"num": 30, "acc": 0.667}, "Music": {"num": 30, "acc": 0.167}, "Overall-Business": {"num": 150, "acc": 0.287}, "Accounting": {"num": 30, "acc": 0.233}, "Economics": {"num": 30, "acc": 0.467}, "Finance": {"num": 30, "acc": 0.133}, "Manage": {"num": 30, "acc": 0.3}, "Marketing": {"num": 30, "acc": 0.3}, "Overall-Science": {"num": 150, "acc": 0.28}, "Biology": {"num": 30, "acc": 0.333}, "Chemistry": {"num": 30, "acc": 0.133}, "Geography": {"num": 30, "acc": 0.4}, "Math": {"num": 30, "acc": 0.2}, "Physics": {"num": 30, "acc": 0.333}, "Overall-Health and Medicine": {"num": 150, "acc": 0.427}, "Basic_Medical_Science": {"num": 30, "acc": 0.4}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.4}, "Pharmacy": {"num": 30, "acc": 0.4}, "Public_Health": {"num": 30, "acc": 0.4}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.7}, "History": {"num": 30, "acc": 0.633}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.567}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.324}, "Agriculture": {"num": 30, "acc": 0.533}, "Architecture_and_Engineering": {"num": 30, "acc": 0.3}, "Computer_Science": {"num": 30, "acc": 0.367}, "Electronics": {"num": 30, "acc": 0.133}, "Energy_and_Power": {"num": 30, "acc": 0.4}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.396}} ``` **gemma-4-12B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.7}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.767}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.747}, "Accounting": {"num": 30, "acc": 0.767}, "Economics": {"num": 30, "acc": 0.767}, "Finance": {"num": 30, "acc": 0.633}, "Manage": {"num": 30, "acc": 0.7}, "Marketing": {"num": 30, "acc": 0.867}, "Overall-Science": {"num": 150, "acc": 0.647}, "Biology": {"num": 30, "acc": 0.6}, "Chemistry": {"num": 30, "acc": 0.633}, "Geography": {"num": 30, "acc": 0.567}, "Math": {"num": 30, "acc": 0.6}, "Physics": {"num": 30, "acc": 0.833}, "Overall-Health and Medicine": {"num": 150, "acc": 0.68}, "Basic_Medical_Science": {"num": 30, "acc": 0.667}, "Clinical_Medicine": {"num": 30, "acc": 0.633}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.267}, "Pharmacy": {"num": 30, "acc": 0.833}, "Public_Health": {"num": 30, "acc": 1.0}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.817}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.9}, "Sociology": {"num": 30, "acc": 0.8}, "Psychology": {"num": 30, "acc": 0.767}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.6}, "Agriculture": {"num": 30, "acc": 0.467}, "Architecture_and_Engineering": {"num": 30, "acc": 0.667}, "Computer_Science": {"num": 30, "acc": 0.733}, "Electronics": {"num": 30, "acc": 0.567}, "Energy_and_Power": {"num": 30, "acc": 0.667}, "Materials": {"num": 30, "acc": 0.567}, "Mechanical_Engineering": {"num": 30, "acc": 0.533}, "Overall": {"num": 900, "acc": 0.683}} ``` **gemma-4-31B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.667}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.8}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.573}, "Accounting": {"num": 30, "acc": 0.633}, "Economics": {"num": 30, "acc": 0.733}, "Finance": {"num": 30, "acc": 0.433}, "Manage": {"num": 30, "acc": 0.533}, "Marketing": {"num": 30, "acc": 0.533}, "Overall-Science": {"num": 150, "acc": 0.527}, "Biology": {"num": 30, "acc": 0.667}, "Chemistry": {"num": 30, "acc": 0.567}, "Geography": {"num": 30, "acc": 0.5}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.633}, "Overall-Health and Medicine": {"num": 150, "acc": 0.673}, "Basic_Medical_Science": {"num": 30, "acc": 0.733}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.467}, "Pharmacy": {"num": 30, "acc": 0.8}, "Public_Health": {"num": 30, "acc": 0.833}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.825}, "History": {"num": 30, "acc": 0.833}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.767}, "Psychology": {"num": 30, "acc": 0.833}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.405}, "Agriculture": {"num": 30, "acc": 0.667}, "Architecture_and_Engineering": {"num": 30, "acc": 0.2}, "Computer_Science": {"num": 30, "acc": 0.567}, "Electronics": {"num": 30, "acc": 0.333}, "Energy_and_Power": {"num": 30, "acc": 0.533}, "Materials": {"num": 30, "acc": 0.3}, "Mechanical_Engineering": {"num": 30, "acc": 0.233}, "Overall": {"num": 900, "acc": 0.589}} ``` **gemma-4-26B-A4B-it** ```json Config theme={null} {"Overall-Art and Design": {"num": 120, "acc": 0.717}, "Art": {"num": 30, "acc": 0.733}, "Art_Theory": {"num": 30, "acc": 0.833}, "Design": {"num": 30, "acc": 0.867}, "Music": {"num": 30, "acc": 0.433}, "Overall-Business": {"num": 150, "acc": 0.493}, "Accounting": {"num": 30, "acc": 0.533}, "Economics": {"num": 30, "acc": 0.533}, "Finance": {"num": 30, "acc": 0.333}, "Manage": {"num": 30, "acc": 0.5}, "Marketing": {"num": 30, "acc": 0.567}, "Overall-Science": {"num": 150, "acc": 0.473}, "Biology": {"num": 30, "acc": 0.633}, "Chemistry": {"num": 30, "acc": 0.367}, "Geography": {"num": 30, "acc": 0.533}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.567}, "Overall-Health and Medicine": {"num": 150, "acc": 0.62}, "Basic_Medical_Science": {"num": 30, "acc": 0.767}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.433}, "Pharmacy": {"num": 30, "acc": 0.7}, "Public_Health": {"num": 30, "acc": 0.667}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.758}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.833}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.667}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.376}, "Agriculture": {"num": 30, "acc": 0.633}, "Architecture_and_Engineering": {"num": 30, "acc": 0.367}, "Computer_Science": {"num": 30, "acc": 0.533}, "Electronics": {"num": 30, "acc": 0.167}, "Energy_and_Power": {"num": 30, "acc": 0.367}, "Materials": {"num": 30, "acc": 0.367}, "Mechanical_Engineering": {"num": 30, "acc": 0.2}, "Overall": {"num": 900, "acc": 0.549}} ``` #### ASR
Model WER Avg Latency (s) Throughput (req/s)
gemma-4-E2B-it 23.86% 0.212 2.99
gemma-4-E4B-it 29.55% 0.366 2.46
gemma-4-12B-it Supported (see §4.5) — —
gemma-4-31B-it Not Supported — —
gemma-4-26B-A4B-it Not Supported — —
#### FLEUR (EN\_US)
Model WER Avg Latency (s) Throughput (req/s)
gemma-4-E2B-it 7.37% 0.8963s 16.25
gemma-4-E4B-it 6.08% 0.8707s 16.20
gemma-4-12B-it Supported (see §4.5) — —
gemma-4-31B-it Not Supported — —
gemma-4-26B-A4B-it Not Supported — —
### 5.3 Logits correctness validation **gemma-4-E2B-it** ```shell Command theme={null} $ python -m sglang.bench_one_batch --correct --model google/gemma-4-E2B-it .... prefill logits (final): tensor([[-25.3063, -2.5718, -10.3674, ..., -25.3779, -25.5181, -25.2337]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E2B-it .... prefill logits (final) tensor([-25.3281, -2.1367, -10.2266, ..., -25.4375, -25.5000, -25.2500], device='cuda:0', dtype=torch.float16) .... ``` **gemma-4-E4B-it** ```shell Command theme={null} $ python -m sglang.bench_one_batch --correct --model google/gemma-4-E4B-it .... prefill logits (final): tensor([[-17.6478, 7.9901, -5.6505, ..., -17.5658, -17.6478, -17.7293]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E4B-it .... prefill logits (final) tensor([-17.5625, 8.0469, -5.5742, ..., -17.4688, -17.5625, -17.6719], device='cuda:0', dtype=torch.float16) .... ``` **gemma-4-31B-it** ```shell Command theme={null} $ python -m sglang.bench_one_batch --correct --model google/gemma-4-31B-it .... prefill logits (final): tensor([[-2.0748, 1.1245, -7.4356, ..., -2.1059, -2.1525, -2.2303]], device='cuda:0') .... $ python scripts/playground/reference_hf.py --model-path google/gemma-4-31B-it .... prefill logits (final) tensor([-2.1133, 1.2656, -7.4766, ..., -2.1523, -2.2012, -2.2695], device='cuda:0', dtype=torch.float16) .... ```
# LLaDA 2.1 Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/LLaDA-2.1 ## 1. Model Introduction [LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation. **Key Features:** * **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight * **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores * **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling * **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following * **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model **Available Models:**
Model Parameters Architecture Context Length HuggingFace
**LLaDA2.1-mini** 16B MoE (20 layers, 16 attention heads) 32,768 tokens [inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)
**LLaDA2.1-flash** 100B MoE 32,768 tokens [inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)
**License:** Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs. ### 3.2 Configuration Tips **dLLM-Specific Parameters:**
Parameter Description Recommended Value
`--dllm-algorithm` Diffusion decoding algorithm `JointThreshold`
`--trust-remote-code` Required for LLaDA model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.8`
`--max-running-requests` Maximum concurrent requests `1` (for best quality)
`--attention-backend` Attention computation backend `flashinfer`
**Decoding Mode Comparison:**
Mode Threshold Speed Quality Best For
**Quality Mode (Q)** Conservative Moderate Higher benchmark scores Accuracy-critical tasks
**Speed Mode (S)** Aggressive Very fast, relies on T2T editing Slightly lower Throughput-critical tasks
**Hardware Requirements:** * **LLaDA2.1-mini (16B)**: \~47 GB VRAM, runs on a single GPU (TP=1) * **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200) ## 4. Model Invocation ### 4.1 Deployment Start the server using the command generated above, for example: ```shell Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-mini \ --dllm-algorithm JointThreshold \ --tp 1 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer \ --host 0.0.0.0 \ --port 8000 ``` ### 4.2 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Explain what a diffusion language model is in simple terms."} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Sure! Let's break it down in simple terms. A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text. Here’s how it works, using a simple real-life analogy: Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together. Now, think of the process like this: 1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper). 2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book. 3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start. 4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word. So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story. In short: A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned. ``` ### 4.3 Advanced Usage #### 4.3.1 Streaming ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ````text Output theme={null} Here are several ways to implement the Fibonacci sequence in Python: ## 1. Recursive Approach (Simple but Inefficient) ```python def fibonacci_recursive(n): """ Compute the nth Fibonacci number using recursion. Args: n (int): The position in the Fibonacci sequence (0-indexed) Returns: int: The nth Fibonacci number Raises: ValueError: If n is negative """ if n < 0: raise ValueError("n must be non-negative") if n <= 1: return n return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2) # Example usage print(fibonacci_recursive(10)) # Output: 55 ``` ## 2. Iterative Approach (Efficient) ... ```` #### 4.3.2 Code Generation ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/LLaDA2.1-mini", messages=[ {"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."} ], max_tokens=2048 ) print(response.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} ```python def is_palindrome(s): """ Check if a string is a palindrome. A palindrome is a word, phrase, or sequence that reads the same backward as forward. This function ignores case, spaces, punctuation, and non characters characters. Args: s (str): The string to check Returns: bool: True if the string is a palindrome, False otherwise Examples: >>> is_palindrome("racecar") True >>> is_palindrome("A man a plan a canal Panama") True >>> is_palindrome("race a car") False >>> is_palindrome("") True >>> is_palindrome("a") True """ # Remove non-alphanumeric characters and convert to lowercase cleaned = ''.join(char.lower() for char in s if char.isalnum()) # Check if the cleaned string reads the same forwards and backwards return cleaned == cleaned[::-1] # Test cases def test_is_palindrome(): """Test the is_palindrome function with various inputs.""" # Test basic palindromes assert is_palindrome("racecar") == True assert is_palindrome("level") == True assert is_palindrome("madam") == True assert is_palindrome("radar") == True # Test palindromes with spaces and punctuation assert is_palindrome("A man a plan a canal Panama") == True assert is_palindrome("race a car") == False assert is_palindrome("Was it a car or a cat I saw?") == True assert is_palindrome("Madam, I'm Adam") == True # Test edge cases assert is_palindrome("") == True assert is_palindrome("a") == True assert is_palindrome("A") == True assert is_palindrome("Aa") == True # Test non-palindromes assert is_palindrome("hello") == False assert is_palindrome("world") == False assert is_palindrome("python") == False # Test single characters assert is_palindrome("1") == True assert is_palindrome("1") == True print("All tests passed!") # Run the tests if __name__ == "__main__": # Example usage print("Testing isalindrome function:") print(f"'racecar' {is_palindrome('racecar')}") print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}") print(f"'race a car': {is_palindrome('race a car')}") print(f"'hello': {is_palindrome('hello')}") # Run tests test_is_palindrome() ``` This implementation includes: 1. **Comprehensive function** `is_palindrome()` that: - Ignores case by converting to lowercase - Removes all non-alphanumeric characters (spaces, punctuation, etc.) - Uses string slicing (`[::-1]`) to reverse the string 2. **Detailed docstring** explaining: - What the function does - How it works - Return value - Examples of usage 3. **Extensive test cases** covering: - Basic palindromes - Palindromes with spaces and punctuation - Edge cases (empty string, single character) - Non-palindromes - Mixed case scenarios 4. **Test function** that uses assertions to verify the function works correctly The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama". ```` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 (4x) * SGLang Version: 0.5.8+ #### 5.1.1 LLaDA2.1-mini **Model Deployment:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-mini \ --dllm-algorithm JointThreshold \ --tp 1 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer ``` * Latency Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-mini \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Latency Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 9.90 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3433 Request throughput (req/s): 1.01 Input token throughput (tok/s): 616.26 Output token throughput (tok/s): 426.26 Peak output token throughput (tok/s): 1010.00 Peak concurrent requests: 3 Total token throughput (tok/s): 1042.53 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 988.87 Median E2E Latency (ms): 655.27 P90 E2E Latency (ms): 1952.50 P99 E2E Latency (ms): 2932.19 ---------------Time to First Token---------------- Mean TTFT (ms): 152.74 Median TTFT (ms): 150.37 P99 TTFT (ms): 229.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.16 Median TPOT (ms): 2.08 P99 TPOT (ms): 3.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.10 Median ITL (ms): 1.99 P95 ITL (ms): 4.03 P99 ITL (ms): 6.34 Max ITL (ms): 26.59 ================================================== ``` * Throughput Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-mini \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Throughput Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 467.74 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 189717 Request throughput (req/s): 1.07 Input token throughput (tok/s): 534.12 Output token throughput (tok/s): 540.17 Peak output token throughput (tok/s): 1753.00 Peak concurrent requests: 105 Total token throughput (tok/s): 1074.30 Concurrency: 90.77 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 84912.27 Median E2E Latency (ms): 86564.26 P90 E2E Latency (ms): 110567.26 P99 E2E Latency (ms): 114303.38 ---------------Time to First Token---------------- Mean TTFT (ms): 83920.39 Median TTFT (ms): 85669.54 P99 TTFT (ms): 112969.91 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.67 Median TPOT (ms): 1.65 P99 TPOT (ms): 4.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.69 Median ITL (ms): 1.46 P95 ITL (ms): 3.96 P99 ITL (ms): 4.84 Max ITL (ms): 92.08 ================================================== ``` #### 5.1.2 LLaDA2.1-flash **Model Deployment:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.1-flash \ --dllm-algorithm JointThreshold \ --tp 4 \ --trust-remote-code \ --mem-fraction-static 0.8 \ --max-running-requests 1 \ --attention-backend flashinfer ``` * Latency Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Latency Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 14.46 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3276 Request throughput (req/s): 0.69 Input token throughput (tok/s): 421.79 Output token throughput (tok/s): 291.75 Peak output token throughput (tok/s): 676.00 Peak concurrent requests: 3 Total token throughput (tok/s): 713.53 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1445.16 Median E2E Latency (ms): 968.06 P90 E2E Latency (ms): 3101.86 P99 E2E Latency (ms): 4208.49 ---------------Time to First Token---------------- Mean TTFT (ms): 231.63 Median TTFT (ms): 242.67 P99 TTFT (ms): 341.33 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.04 Median TPOT (ms): 2.79 P99 TPOT (ms): 5.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 3.05 Median ITL (ms): 2.41 P95 ITL (ms): 7.25 P99 ITL (ms): 8.27 Max ITL (ms): 29.27 ================================================== ``` * Throughput Benchmark ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model inclusionAI/LLaDA2.1-flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Throughput Result**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 671.85 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 177961 Request throughput (req/s): 0.74 Input token throughput (tok/s): 371.85 Output token throughput (tok/s): 376.07 Peak output token throughput (tok/s): 1521.00 Peak concurrent requests: 103 Total token throughput (tok/s): 747.92 Concurrency: 91.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 122658.36 Median E2E Latency (ms): 125265.55 P90 E2E Latency (ms): 159554.07 P99 E2E Latency (ms): 165174.88 ---------------Time to First Token---------------- Mean TTFT (ms): 121009.17 Median TTFT (ms): 124437.80 P99 TTFT (ms): 163579.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.73 Median TPOT (ms): 2.16 P99 TPOT (ms): 7.13 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.38 Median ITL (ms): 1.40 P95 ITL (ms): 6.89 P99 ITL (ms): 8.60 Max ITL (ms): 176.78 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark ```bash Command theme={null} python -m sglang.test.few_shot_gsm8k \ --num-questions 200 \ --port 8000 ``` **Results:** ```text Output theme={null} Accuracy: 0.895 Invalid: 0.000 Latency: 100.552 s Output throughput: 262.094 token/s ``` # Ling-2.5-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.5-1T ## 1. Model Introduction [Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality. **Key Features:** * **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens. * **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume \~4x the output tokens. * **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following. * **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4. * **Context Length**: 256K -> 1M (YaRN) **Available Models:** * **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) **License:** MIT ## 2. SGLang Installation Ling-2.5-1T runs on the standard SGLang Docker image: ```bash Command theme={null} # NVIDIA (H200 / B200 / GB200 / GB300) docker pull lmsysorg/sglang:latest ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/install) guide for setup instructions. ## 3. Model Deployment Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform. ### Configuration Tips * The `--trust-remote-code` flag is required for this model due to custom modeling code. * `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes. * Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading. * On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details). ## 4. Model Invocation ### 4.1 Basic Usage For example, launch the server on 2 H200 nodes: ```bash Command theme={null} export MASTER_IP=10.10.0.1 # The IP of Node 0 export PORT=30000 export DIST_PORT=50000 # Node 0: python3 -m sglang.launch_server \ --model-path inclusionAI/Ling-2.5-1T \ --trust-remote-code \ --tp-size 8 \ --pp-size 2 \ --nnodes 2 \ --node-rank 0 \ --host 0.0.0.0 \ --port ${PORT} \ --dist-init-addr ${MASTER_IP}:${DIST_PORT} \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \ --mem-frac 0.95 # Node 1: python3 -m sglang.launch_server \ --model-path inclusionAI/Ling-2.5-1T \ --trust-remote-code \ --tp-size 8 \ --pp-size 2 \ --nnodes 2 \ --node-rank 1 \ --dist-init-addr ${MASTER_IP}:${DIST_PORT} \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \ --mem-frac 0.95 ``` Once the server is running, send requests to the master node: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` Output: ```json Config theme={null} { "id": "e82af153da844ee6aed7a27a3187f2f4", "object": "chat.completion", "created": 1771216764, "model": "auto", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is **Paris**.\n\n**Additional details:**\n* It is the largest city in France.\n* It is located in the north-central part of the country along the Seine River.\n* Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).", "reasoning_content": null, "tool_calls": null }, "logprobs": null, "finish_reason": "stop", "matched_stop": 156895 } ], "usage": { "prompt_tokens": 25, "total_tokens": 93, "completion_tokens": 68, "prompt_tokens_details": null, "reasoning_tokens": 0 } } ``` For more API usage examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Tool Calling Example ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-2.5-1T", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` Output: ```json Config theme={null} { "id": "b968e45c7d414f7482c8ffc0f9c6b688", "object": "chat.completion", "created": 1771216520, "model": "inclusionAI/Ling-2.5-1T", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "reasoning_content": null, "tool_calls": [ { "id": "call_e75f711d8ad840ed9d382c9e", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI\"}" } } ] }, "logprobs": null, "finish_reason": "tool_calls", "matched_stop": null } ], "usage": { "prompt_tokens": 173, "total_tokens": 196, "completion_tokens": 23, "prompt_tokens_details": null, "reasoning_tokens": 0 } } ``` ## 5. Benchmark ### GSM8K * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py ``` * Test Result ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 45.410 s Output throughput: 560.642 token/s ``` # Ling-2.6 Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.6 ## 1. Model Introduction The **Ling-2.6** family from inclusionAI is the next iteration of the Ling instant-model series. Continuing the architectural direction set by Ling-2.5, Ling-2.6 doubles down on **inference efficiency**, **token efficiency**, and **agent performance** — staying competitive with frontier instant models while being faster, leaner, and better suited for production agent workloads. **Key Features:** * **Hybrid Linear Attention**: A `1:7 MLA + Lightning Linear` hybrid built on top of a highly sparse MoE backbone. Compared with same-class SOTA models, Ling-2.6-flash shows up to \~4× higher prefill and decode throughput in long-context scenarios; Ling-2.6-1T is shipped in FP8 so it fits a single GB300 node with `--tp 4`. * **Token Efficiency**: Trained with explicit token-efficiency objectives. On the full Artificial Analysis suite, Ling-2.6-flash uses only \~15M output tokens while remaining competitive — a meaningfully stronger intelligence-per-token profile than long-reasoning peers. * **Agentic Capabilities**: Refined for tool use, multi-step planning, and long-horizon execution. Reaches SOTA-class results on **BFCL-V4**, **TAU2-bench**, **SWE-bench Verified**, **Claw-Eval**, and **PinchBench**, and is validated against Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw. * **Long Context**: Native 128K, extendable to **256K (Ling-2.6-flash)** and **256K → 1M (Ling-2.6-1T via YaRN)**. **Available Models:** * **BF16**: [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) — 104B total / 7.4B active * **FP8 (E4M3)**: [inclusionAI/Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) — \~1T total **License:** MIT ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment ### 3.1 Ling-2.6-flash Ling-2.6-flash is a 104B/7.4B-active MoE that runs comfortably on a single 4-GPU node. Use the selector below to generate the launch command for your hardware. #### Configuration Tips * `--trust-remote-code` is required (custom `BailingMoeV2_5ForCausalLM` modeling code). * `--tp-size 4` is the reference layout. On 4× H20-3e the model reaches \~340 tokens/s decode at TP=4, batch 32. * Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you. * `--tool-call-parser qwen25` matches the model's `...` schema. * The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `...` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode). * **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example. ### 3.2 Ling-2.6-1T Ling-2.6-1T ships in **FP8 (E4M3)**, so unlike Ling-2.5-1T it fits a **single GB300 node with `--tp 4`**. On smaller GPUs (H200/B200), a 2-node deployment with `--pp-size 2` is required. #### Configuration Tips * `--trust-remote-code` is required for the custom modeling code. * `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` significantly speeds up the multi-shard FP8 weight load (26 safetensors shards + an MTP layer). * Use `--tool-call-parser qwen` for tool calling. * The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6's chat template defaults to `detailed thinking off`, while SGLang's `qwen3` reasoning parser assumes default-thinking semantics — combining the two requires a per-request workaround for tool calls (see [§4.3 Thinking Mode](#4-3-thinking-mode)). Only enable `--reasoning-parser qwen3` if you specifically want `...` blocks split into `reasoning_content`. * For 2-node deployments, set `MASTER_IP`, `PORT`, and `DIST_PORT` consistently across both nodes. ## 4. Model Invocation For example, launch a Ling-2.6-1T server on a single GB300 node: ```bash Command theme={null} sglang serve \ --model-path inclusionAI/Ling-2.6-1T \ --tp-size 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 \ --tool-call-parser qwen \ --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' ``` ### 4.1 Basic Usage ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}' ``` Output: ```json Config theme={null} { "id": "...", "object": "chat.completion", "model": "auto", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "The capital of France is **Paris**.", "reasoning_content": null, "tool_calls": null }, "finish_reason": "stop" } ] } ``` ### 4.2 Tool Calling Example ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` Output: ```json Config theme={null} { "choices": [ { "message": { "role": "assistant", "content": null, "tool_calls": [ { "id": "call_...", "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` ### 4.3 Thinking Mode Both Ling-2.6-flash and Ling-2.6-1T are **controllable-reasoning** models. Their chat template uses textual directives in the system message — `detailed thinking on` or `detailed thinking off` — to toggle thinking. The template **defaults to `detailed thinking off`** when neither phrase is present, and it does **not** read the Qwen3-style `enable_thinking` template variable. #### Enabling thinking Include `detailed thinking on` in the first system message: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [ {"role": "system", "content": "detailed thinking on"}, {"role": "user", "content": "If a box has 12 red balls and 8 blue balls, then 5 red balls are removed, how many balls remain?"} ] }' ``` If you already have a system prompt, append the directive on its own line: ```json theme={null} {"role": "system", "content": "You are a helpful assistant.\ndetailed thinking on"} ``` When thinking is on, the model emits `...` blocks before its final answer. To get those split into `message.reasoning_content` automatically, also launch the server with `--reasoning-parser qwen3`. #### Caveat: `--reasoning-parser qwen3` + tool calling The SGLang `qwen3` reasoning parser was written for Qwen3, where models are **default-thinking** and clients opt out via `chat_template_kwargs.enable_thinking=false`. Ling-2.6 is the opposite — default-non-thinking, with toggling done in the system message. As a result, when the server is launched with **both** `--tool-call-parser qwen` and `--reasoning-parser qwen3`, every tool-call request must include `chat_template_kwargs.enable_thinking=false`, otherwise the parser routes the `...` block into `reasoning_content` instead of `message.tool_calls`: ```bash Command theme={null} curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [...], "tool_choice": "auto", "chat_template_kwargs": {"enable_thinking": false} }' ``` `enable_thinking` here is consumed by the SGLang reasoning parser, **not** by the chat template — Ling-2.6's template ignores it. For the simplest configuration, just omit `--reasoning-parser qwen3` and toggle thinking via the system message. For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request). ## 5. Benchmark ### GSM8K (Ling-2.6-1T, GB300 × 4) Reference run on a single GB300 node with `--tp 4`: ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py ``` ```text Output theme={null} Accuracy: 0.9621 (1269 / 1319) ``` For Ling-2.6-flash, see the official numbers on the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash) (BFCL-V4, TAU2-bench, SWE-bench Verified, Claw-Eval, PinchBench, Artificial Analysis). # Ling-3.0-flash Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash Deploy Ling-3.0-flash with SGLang — a 124B hybrid KDA + MLA MoE in BF16, FP8, INT4, or MXFP4 on Hopper and Blackwell GPUs. ## Deployment ```bash Command theme={null} docker pull lmsysorg/sglang:dev-Ling-3.0-flash ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. Three serving strategies are covered: * **Low-Latency** — fastest reply for a single user. Pick for chat. These recipes run speculative decoding: pick **NEXTN** (built-in MTP layer, no extra checkpoint) or **DSPARK** (external draft model) in the **Spec Decode** selector of the Deploy panel. * **High-Throughput** — most tokens per second across many users. Best for batch jobs. These recipes turn speculative decoding off, since at saturation the draft/verify overhead outweighs the speedup. * **HiCache + Mooncake** — writes reusable prefixes to Mooncake L3 storage. Start the Mooncake services in §3.3 before launching the generated server command. ## Playground The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction Ling-3.0-flash is a hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family. It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network. This keeps per-token inference cost close to a small model — **124B total parameters with only 5.1B active** — while retaining large-model capacity. It is a hybrid-reasoning model with thinking enabled by default, and it supports structured tool calling. Native context length is 256K. **Available Models:** * **BF16**: [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) — 124B total / 5.1B active * **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-flash-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8) * **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-flash-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4) * **MXFP4**: [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4) * **DSPARK draft**: [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) — 5-layer DSpark draft checkpoint for DSPARK speculative decoding **License:** MIT **Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash). ## 2. Configuration Tips * BF16 tensor parallelism follows the GPU: `--tp 4` on 141 GB-class cards (H20-3e, H200) and 4-GPU Blackwell nodes (B200, GB300); `--tp 8` on 80 GB cards (H100, H800). * The FP8 recipes pair `--tp` with a matching `--ep-size` (`--tp 4 --ep-size 4` on 4-GPU nodes, `--tp 8 --ep-size 8` on H100/H800). The checkpoint uses blockwise (128×128) E4M3 expert weights, so a pure tensor-parallel shard must satisfy `(768 / TP) % 128 == 0` — only TP2 qualifies; expert parallelism splits experts whole instead of by column, which lifts that restriction and uses the full node. SGLang detects the quantization format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed. * INT4 uses compressed-tensors W4A16 experts. With the default MoE runner, SGLang selects Marlin on H200 and the graph-compatible Triton implementation on Blackwell; do not force a backend in the launch command. * MXFP4 uses the native FlashInfer runner on H200 and B200. The checkpoint mixes MXFP4 routed experts with block-FP8 dense and shared projections; on B200, `--fp8-gemm-backend triton` avoids an unsupported FlashInfer FP8 tactic while the routed experts remain native MXFP4. * `--reasoning-parser ling3` and `--tool-call-parser ling3` enable Ling-3.0-specific reasoning and structured tool-call parsing; toggle them in the **Parsers** card of the [Playground](#playground). * Both the chat template and the `ling3` reasoning parser default to thinking on. A single request can turn it off with `"chat_template_kwargs": {"enable_thinking": false}` (see §3.1). * The BF16/FP8 recipes use `--mem-fraction-static 0.8`; INT4/MXFP4 use `0.85`. These values reserve the headroom used by the validated graph-enabled runs. * The checkpoint ships a built-in MTP layer (`num_nextn_predict_layers: 1`); enable it with `--speculative-algorithm NEXTN` — no separate draft model is needed. The Low-Latency recipes default to it via the **Spec Decode** selector; toggle it in the **Speculative Decoding** card of the [Playground](#playground). * DSPARK is the alternative speculative path: it drafts with the external [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) checkpoint instead of the built-in MTP layer. Pick **DSPARK** in the Deploy panel's **Spec Decode** selector (also available in the [Playground](#playground)); it composes `--speculative-algorithm DSPARK`, the draft path, and `--enable-linear-replayssm-spec`, the KDA verify path that folds per-draft intermediate states onto a fixed ring instead of snapshotting full states. The draft block size auto-infers from the checkpoint; tune it with `--speculative-dspark-block-size`. The draft's block size of 8 makes the verify window 9 tokens, and the KDA ReplaySSM ring must be a power of two at least twice the window — the recipe therefore pins `--linear-replayssm-cache-len 32`, since the 16 default fails startup validation. MXFP4 requires `--moe-runner-backend flashinfer_mxfp4`; the mixed checkpoint does not auto-select it. * Native context is 256K; SGLang reads it from the checkpoint's `max_position_embeddings`, so no `--context-length` flag is needed. * The **HiCache** card in the [Playground](#playground) exposes the validated Mooncake L3 path. It adds the hybrid-KDA scheduler and prefix-key settings together; see §3.3 for the required services. ## 3. Advanced Usage ### 3.1 Reasoning Ling-3.0-flash thinks by default. With `--reasoning-parser ling3` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-flash", "messages": [{"role": "user", "content": "What is 15% of 240?"}] }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "15% of 240 = **36**", "reasoning_content": "The user is asking a simple percentage calculation: 15% of 240. This is straightforward: 0.15 × 240 = 36.", "tool_calls": null }, "finish_reason": "stop" } ] } ``` Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. ### 3.2 Tool Calling With `--tool-call-parser ling3` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-flash", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "", "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool with a query about the latest AI news.", "tool_calls": [ { "id": "call_0822dd418aa34254aa5b19e1", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI 2025\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). ### 3.3 HiCache with Mooncake The HiCache recipes use Mooncake as L3 prefix storage. Start the metadata server, master, and storage client before you launch SGLang. The following command runs all three services from the same image in a separate container: ```bash Command theme={null} docker run --rm --network host --ipc=host \ lmsysorg/sglang:dev-Ling-3.0-flash \ bash -lc ' python3 -m mooncake.http_metadata_server --port 8290 & mooncake_master --port 50171 --metrics_port 9024 & exec mooncake_client \ --host=127.0.0.1 \ --port=50172 \ --master_server_address=127.0.0.1:50171 \ --metadata_server=http://127.0.0.1:8290/metadata \ --protocol=tcp \ --device_names= \ --global_segment_size=4294967296 \ --enable_http_server=true \ --http_port=8291 ' ``` Then select **HiCache + Mooncake** in Deployment, or enable **HiCache** in the Playground. Docker commands use host networking so the SGLang container can reach these localhost services. You can change the master and metadata endpoints in the **Env** dialog. This validated setup uses TCP, so `MOONCAKE_DEVICE=` and the client's `--device_names=` are intentionally empty. Set both to your actual device list only when you configure an RDMA deployment. With the default `chunked_prefill_size` of 8192, a cold request writes through only when its uncached extend length fits in one chunk. A longer cold first request skips write-through for that influx; a repeat with the same prefix can hit the device radix cache and proceed normally. For this hybrid KDA model, use the Mooncake L3 recipe shown here. Host-memory L2 eviction is not exposed because the KDA cache path is not currently compatible with it. Track the limitation in [issue #33713](https://github.com/sgl-project/sglang/issues/33713). For storage sizing and backend details, see [HiCache best practices](/docs/advanced_features/hicache_best_practices). # Ling-3.0-tiny Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny Deploy Ling-3.0-tiny with SGLang — a compact ~7.9B total / ~1.2B active hybrid KDA + MLA MoE in BF16, FP8, or INT4, with thinking mode and tool calling. ## Deployment ```bash Command theme={null} docker pull lmsysorg/sglang:dev-Ling-3.0-tiny ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. One serving strategy is covered: * **High-Throughput** — most tokens per second across many users. Best for batch jobs. Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so there is no NEXTN speculative-decoding recipe. ## Playground The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction Ling-3.0-tiny is a compact hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family — the small variant of [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash). It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers on top of a fine-grained MoE feed-forward network, keeping per-token inference cost near a \~1B dense model — **\~7.9B total parameters with \~1.2B active** — while retaining large-model capacity. It is a thinking model with chain-of-thought enabled by default, and it supports structured tool calling. Native context length is 128K. Unlike Ling-3.0-flash, it ships **no built-in MTP draft layer**, so it does not use NEXTN speculative decoding. **Available Models:** * **BF16**: [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — \~7.9B total / \~1.2B active * **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-tiny-fp8](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8) * **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-tiny-int4](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4) **License:** MIT **Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-tiny). ## 2. Configuration Tips * At \~7.9B total / 15.8 GB in BF16 (\~7.9 GB in FP8 and \~5.8 GB in INT4), a single GPU is plenty on every supported card. Tensor parallelism is only useful to raise aggregate KV-cache capacity for many long-context concurrent requests — add `--tp 2`/`--tp 4` to a multi-GPU serve directly. * Use the dedicated `lmsysorg/sglang:dev-Ling-3.0-tiny` runtime image; it includes the compressed-tensors Hopper and Blackwell backends that INT4 needs. * The FP8 checkpoint uses blockwise (128×128) E4M3 weights with dynamic activations, quantized from the BF16 model with attention projections, the dense MoE gate, and the lm\_head left in higher precision. SGLang detects the format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed, and the same single-GPU recipe serves it. * The INT4 checkpoint uses symmetric group-32 W4A16 routed experts. SGLang selects Marlin on Hopper and Triton WNA16 on Blackwell automatically; no explicit quantization or MoE backend flag is needed. * Unlike Ling-3.0-flash (which pairs `--reasoning-parser ling3` / `--tool-call-parser ling3`), Ling-3.0-tiny uses `--reasoning-parser deepseek-r1` and `--tool-call-parser glm45` (its auto-detected template pairing) — the template wraps tool calls in `` blocks and emits an inline `...` chain-of-thought. Toggle them in the **Parsers** card of the [Playground](#playground). * Only `--model-path`, `--host`, and `--port` are needed. SGLang auto-resolves the context length (native 128K from `max_position_embeddings`), the attention backend, and `--mem-fraction-static` from the GPU and the CUDA-graph runtime, so the recipes leave them unset. * The chat template defaults to thinking on. Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` for direct answers without the `...` block. * Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so `--speculative-algorithm NEXTN` is not applicable. ## 3. Advanced Usage ### 3.1 Reasoning With `--reasoning-parser deepseek-r1` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-tiny", "messages": [{"role": "user", "content": "What is 15% of 240?"}] }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "15% of 240 is **36**.\n\n**Calculation:** 0.15 × 240 = 36", "reasoning_content": "The user is asking for 15% of 240. This is a simple percentage calculation.\n\n15% of 240 = 0.15 × 240 = 36\n\nLet me verify: 0.15 × 240 = 0.15 × 200 + 0.15 × 40 = 30 + 6 = 36. Yes, that's correct.", "tool_calls": null }, "finish_reason": "stop" } ] } ``` Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`. ### 3.2 Tool Calling With `--tool-call-parser glm45` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`: ```bash Command theme={null} curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "inclusionAI/Ling-3.0-tiny", "messages": [{"role": "user", "content": "Search for the latest news about AI"}], "tools": [{ "type": "function", "function": { "name": "search", "description": "Search for information on the internet", "parameters": { "type": "object", "properties": { "query": {"type": "string", "description": "The search query"} }, "required": ["query"] } } }], "tool_choice": "auto" }' ``` ```json Output theme={null} { "choices": [ { "message": { "role": "assistant", "content": "Let me search for the latest news about AI for you.", "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool to find recent AI news.", "tool_calls": [ { "id": "call_79b73a89696d4544ac6dd724", "index": 0, "type": "function", "function": { "name": "search", "arguments": "{\"query\": \"latest AI news 2025\"}" } } ] }, "finish_reason": "tool_calls" } ] } ``` For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request). # Ring-2.5-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.5-1T ## 1. Model Introduction [Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities. **Key Features:** * **Trillion-Scale Model**: \~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention) * **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens * **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback * **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training * **Tool Calling**: Supports function calling with XML-style tool call format * **Context Length**: 128K -> 256K (YaRN) **Available Models:** * **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) **License:** MIT ## 2. SGLang Installation Ring-2.5-1T runs on the standard SGLang Docker image: ```bash Command theme={null} # NVIDIA (H200 / B200 / GB200 / GB300) docker pull lmsysorg/sglang:latest # For MI300X/325X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x # For MI355X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. ### 3.2 Configuration Tips * The `--trust-remote-code` flag is required for this model due to custom modeling code. * The model uses FP8 quantization (compressed-tensors format). ## 4. Model Invocation Deploy Ring-2.5-1T with the following command (on H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response. ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="inclusionAI/Ring-2.5-1T", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) if delta.content: print(delta.content, end="", flush=True) print() ```
Output Example ```text Output theme={null} We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution. We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36. But we need to present step by step. Also ensure it's clear. We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36. But any method is fine. We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)." We'll show: 15% = 15/100 = 0.15 Then 0.15 × 240 = 36. Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36. Finally, answer: 36. We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36." We'll present as a final answer: \boxed{36}. However, we need to provide step-by-step solution as per instructions. We'll write a full explanation. We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36. Alr. I think that's it. **Step 1:** Write 15% as a fraction or decimal. \[ 15\% = \frac{15}{100} = 0.15\] **Step 2:** Multiply the number (240) by this fraction/decimal. \[ 240 \times 0.15 = 36\] Alternatively, using the fraction: \[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\] **Conclusion:** 15% of 240 is 36. \[ \boxed{36} \] ```
#### 4.2.2 Tool Calling To enable tool calling, add `--tool-call-parser qwen` when launching the server. ```shell Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp 8 \ --trust-remote-code \ --tool-call-parser qwen \ --host 0.0.0.0 \ --port 30000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="inclusionAI/Ring-2.5-1T", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools ) print(response.choices[0].message.tool_calls) ``` **Output Example:** ```text Output theme={null} [ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ## 5. Benchmark ### GSM8K * Deployment Command ```bash Command theme={null} sglang serve \ --model-path inclusionAI/Ring-2.5-1T \ --tp-size 8 \ --trust-remote-code ``` * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --temperature 1.2 --top-p 0.8 --max-new-tokens 32768 --num-questions 200 --tokenizer-path inclusionAI/Ring-2.5-1T --enable-thinking ``` * Test Result ```text Output theme={null} Accuracy: 0.955 Invalid: 0.010 Latency: 615.833 s Output throughput: 412.360 token/s ``` # Ring-2.6-1T Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.6-1T ## 1. Model Introduction [Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is InclusionAI's trillion-parameter flagship reasoning model for real-world complex task execution. It targets agent workflows, engineering development, scientific research analysis, enterprise automation, and other long-horizon settings where the model must plan, use tools, recover from intermediate errors, and keep context across multiple steps. **Key Features:** * **Trillion-Scale Reasoning Model**: `BailingMoeV2_5ForCausalLM` with a `bailing_hybrid` architecture, 80 hidden layers, 256 routed experts, 8 selected experts per token, and FP8 compressed-tensors weights. * **Agent Execution**: Designed for multi-step task decomposition, tool collaboration, context continuation, and long-horizon execution. The model card reports 87.60 on PinchBench, 63.82 on ClawEval, and 95.32 on Tau2-Bench Telecom for the `high` setting. * **Reasoning Effort**: The model card describes `high` and `xhigh` reasoning-effort modes. In SGLang's OpenAI-compatible chat API, use top-level `reasoning_effort: "high"` for production agent workflows. To request the model-card `xhigh` prompt path, pass it through `chat_template_kwargs.reasoning_effort`. * **Hybrid Attention**: Uses the Bailing hybrid stack with MLA plus Lightning linear attention kernels in SGLang. * **Context Length**: Native 128K in the released config. Configure YaRN separately if you need a 256K deployment. **Available Models:** * **FP8 (E4M3 compressed-tensors)**: [inclusionAI/Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) **License:** MIT ## 2. SGLang Installation Ring-2.6-1T requires recent SGLang builds with Bailing hybrid model support. Start with the latest SGLang Docker image when validating this cookbook: ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment Use the selector below to generate a single-node command for the tested hardware targets. ### Configuration Tips * `--trust-remote-code` is required for the model's custom Bailing hybrid implementation. * Use `--tp-size 4` on a single 4-GPU GB300 node. * Use `--tp-size 8` on a single 8-GPU B200 node. * Use `--tp-size 8` on a single 8-GPU H200 node. * Use `--mem-fraction-static 0.95` on GB300 x4. The model uses about 238.5GB/GPU after loading, so lower values can fail during KV-pool initialization. * Use `--mem-fraction-static 0.8` on B200 x8. * Use `--mem-fraction-static 0.95` on H200 x8. * `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` is recommended because the model has 175 large safetensors shards. * Keep `--tool-call-parser glm` enabled by default for OpenAI-compatible tool calls. Ring's template emits XML `/` tool calls, which the `qwen` parser does not convert into `message.tool_calls`. * Keep `--reasoning-parser deepseek-r1` enabled by default so `...` content is split into `message.reasoning_content`. ## 4. Model Invocation ### 4.1 Basic Usage For example, launch the server on a single 4-GPU GB300 node: ```bash Command theme={null} export PORT=30000 sglang serve \ --model-path inclusionAI/Ring-2.6-1T \ --tp-size 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port ${PORT} \ --mem-fraction-static 0.95 \ --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' \ --tool-call-parser glm \ --reasoning-parser deepseek-r1 ``` Send a basic chat request: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 128 }' ``` ### 4.2 Reasoning Effort Ring-2.6-1T exposes two reasoning-effort levels in the model card: `high` and `xhigh`. In SGLang's OpenAI-compatible chat API, start with top-level `reasoning_effort: "high"` for agent and production workflows: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}], "reasoning_effort": "high", "max_tokens": 512 }' ``` For the model-card `xhigh` path, pass the template value explicitly: ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}], "chat_template_kwargs": {"reasoning_effort": "xhigh"}, "max_tokens": 512 }' ``` With the default deployment command, thinking text is separated into `message.reasoning_content` when the model emits `...` blocks. ### 4.3 Tool Calling Example ```bash Command theme={null} curl -s http://localhost:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "auto", "messages": [{"role": "user", "content": "What is the weather in Beijing?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"} }, "required": ["location"] } } }], "tool_choice": "auto", "max_tokens": 512 }' ``` For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request). ## 5. Benchmark ### 5.1 Speed Benchmark * Hardware: NVIDIA B200 GPU (8x), NVIDIA H200 GPU (8x), and NVIDIA GB300 GPU (4x) * Model: `inclusionAI/Ring-2.6-1T` * Docker image: `lmsysorg/sglang:latest` * SGLang version tested: `0.5.11` * Tensor Parallelism: 8 on B200 x8 and H200 x8, 4 on GB300 x4 Use the deployment command from [Section 3](#3-model-deployment), then confirm that the server is healthy before running benchmarks: ```bash Command theme={null} curl -s http://localhost:${PORT}/health curl -s http://localhost:${PORT}/v1/models ``` #### 5.1.1 Latency-Sensitive Benchmark * Test Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port ${PORT} \ --model inclusionAI/Ring-2.6-1T \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results (B200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 207.18 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.05 Input token throughput (tok/s): 29.45 Output token throughput (tok/s): 20.37 Total token throughput (tok/s): 49.82 Mean E2E Latency (ms): 20715.16 Mean TTFT (ms): 187.86 Mean TPOT (ms): 44.65 Mean ITL (ms): 48.76 ================================================== ``` * Test Results (GB300 x4): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 62.21 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.16 Input token throughput (tok/s): 98.07 Output token throughput (tok/s): 67.83 Total token throughput (tok/s): 165.91 Mean E2E Latency (ms): 6218.57 Mean TTFT (ms): 233.04 Mean TPOT (ms): 14.21 Mean ITL (ms): 14.22 ================================================== ``` * Test Results (H200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 57.10 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.18 Input token throughput (tok/s): 106.85 Output token throughput (tok/s): 73.91 Total token throughput (tok/s): 180.76 Mean E2E Latency (ms): 5707.72 Mean TTFT (ms): 163.35 Mean TPOT (ms): 13.17 Mean ITL (ms): 13.17 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Test Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port ${PORT} \ --model inclusionAI/Ring-2.6-1T \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 100 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results (B200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 46.30 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 2.16 Input token throughput (tok/s): 1092.10 Output token throughput (tok/s): 1132.77 Total token throughput (tok/s): 2224.86 Mean E2E Latency (ms): 27581.74 Mean TTFT (ms): 1710.53 Mean TPOT (ms): 51.27 Mean ITL (ms): 49.43 ================================================== ``` * Test Results (GB300 x4): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 55.80 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 1.79 Input token throughput (tok/s): 906.10 Output token throughput (tok/s): 939.84 Total token throughput (tok/s): 1845.94 Mean E2E Latency (ms): 33736.85 Mean TTFT (ms): 2156.40 Mean TPOT (ms): 63.09 Mean ITL (ms): 60.33 ================================================== ``` * Test Results (H200 x8): ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 100 Benchmark duration (s): 44.51 Total input tokens: 50561 Total generated tokens: 52444 Request throughput (req/s): 2.25 Input token throughput (tok/s): 1135.88 Output token throughput (tok/s): 1178.18 Total token throughput (tok/s): 2314.06 Mean E2E Latency (ms): 27177.14 Mean TTFT (ms): 2173.08 Mean TPOT (ms): 51.11 Mean ITL (ms): 47.77 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```bash Command theme={null} python3 -m sglang.test.run_eval \ --eval-name gsm8k \ --host 127.0.0.1 \ --port ${PORT} \ --model auto \ --num-examples 200 \ --num-threads 64 \ --max-tokens 2048 \ --reasoning-effort high ``` * Test Results (B200 x8): ```text Output theme={null} Total latency: 100.378 s Score: 0.990 Output throughput: 627.401 token/s ``` * Test Results (GB300 x4): ```text Output theme={null} Total latency: 98.386 s Score: 0.990 Output throughput: 621.469 token/s ``` * Test Results (H200 x8): ```text Output theme={null} Total latency: 76.849 s Score: 0.990 Output throughput: 793.125 token/s ``` # Intern-S1 Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S1 ## 1. Model Introduction Intern-S1 includes the large **Intern-S1** MoE model and the smaller **Intern-S1-mini** dense model. The command generator below covers BF16 and FP8 serving on NVIDIA H100/H200/B200/B300 platforms. ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install from source: ```bash Command theme={null} uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` ## 3. Model Deployment ### 3.1 Basic Configuration ### 3.2 Configuration Tips * FP8 checkpoints use the matching BF16 checkpoint as tokenizer path. * B300 deployments use `--attention-backend flashinfer`. * Enable `--reasoning-parser interns1` and `--tool-call-parser interns1` when your workload needs structured reasoning or tool-call parsing. # Intern-S2-Mobius Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Mobius Deploy the BF16 and FP8 Intern-S2-Mobius checkpoints with SGLang on NVIDIA H200 and B200. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev ``` Intern-S2-Mobius was upstreamed in PR [#33691](https://github.com/sgl-project/sglang/pull/33691) (merged 2026-08-08) — it lives on `lmsysorg/sglang:dev` (nightly) until the next release cut. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. The two serving strategies cover the main operating points: * **Low-Latency** — MTP (NEXTN) speculative decoding on. Fastest reply for a single user. * **High-Throughput** — spec off, more tokens per second when many users share the server. Speed and accuracy numbers apply to the BF16 checkpoint. They were measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass\@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top\_p 0.95, top\_k 20). The FP8 H200 recipes have launch, text, vision, and stop-reason verification but no benchmark numbers yet. All B200 recipes are inferred and unverified. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Intern-S2-Mobius** is InternLM's 35B scientific multimodal foundation model built on the **Mobius-v0** architecture (continually pre-trained from Qwen3.5-35B, then SFT and RL post-trained). Instead of binding knowledge storage and reasoning computation layer by layer as conventional Transformers do, Mobius organizes knowledge into a **globally shared Memory** that multiple **Reasoners** iteratively query against, yielding two native capabilities: * **Backward Residual Connection** — shallow and deep reasoning stages can reach knowledge across the model rather than relying only on forward layer-wise flow. * **Dynamic Latent Reasoning** — recurrent latent iteration refines hidden states before decoding, internalizing part of the deliberation process and shrinking visible chain-of-thought. The reported result is roughly a **4× end-to-end inference speedup** over the Qwen3.5-35B baseline while holding comparable scores on general reasoning benchmarks and improving on scientific tasks (Biology-Instructions, Mol-Instructions, MolecularIQ). On the serving side the model is a hybrid: 30 of 40 transformer layers use **GDN (Gated Delta Net) linear attention** (`kimi-linear`-family), with a **full-attention** layer every 4th layer (`full_attention_interval: 4` → **10 full-attention** layers), and the bottom of the stack is MoE-routed (2,560 routed experts × 512 intermediate, 8 active per token); a separate **MoE-256 / top-8 MTP (NEXTN) layer** feeds speculative decoding. It takes images via a vision tower and recognizes the standard `<|vision_start|>…<|vision_end|>` + `<|image_pad|>` markers. Context length is **262,144** tokens.
Variant Architecture Context License
Intern-S2-Mobius Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · BF16 262,144 Apache-2.0
Intern-S2-Mobius-FP8 Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · FP8 E4M3 262,144 Apache-2.0
**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Both checkpoints ship `generation_config.json` with different values (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them. **Resources:** [HuggingFace (BF16)](https://huggingface.co/internlm/Intern-S2-Mobius) · [HuggingFace (FP8)](https://huggingface.co/internlm/Intern-S2-Mobius-FP8) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius). ## 2. Configuration Tips * **Trust remote code is required.** Intern-S2-Mobius ships a custom `configuration_interns2_mobius.py` / `modeling_interns2_mobius.py` on its HF repo; every recipe adds `--trust-remote-code`. * **FP8 uses one H200.** The FP8 checkpoint stores E4M3 weights with 128×128 block scales and dynamic activation quantization. The verified H200 recipes use TP=1, `deep_gemm`, `--mem-fraction-static 0.6`, disabled prefill CUDA graphs, and decode graphs up to batch size 16. Higher memory fractions or the default graph range exhausted H200 memory during DeepGEMM warmup; the running MTP server used about 99 GB. * **Speculative decoding schedule.** The checkpoint ships one MTP layer. Enable MTP for the lowest latency (`--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`). We measured accept-length \~3.9/4 draft tokens at 8K-in / 1K-out, roughly tripling single-stream decode speed (median TPOT 9.79 ms → 3.13 ms at conc=1, 14.26 ms → 6.84 ms at conc=16) and roughly doubling mid-concurrency total throughput (9358 → 18029 tokens/s at conc=16, 21395 → 26033 tokens/s at conc=64). The high-throughput recipe stays spec-off because once you can batch wide, its saturation point is higher (34786 tokens/s at conc=256 vs the spec recipe's peak at conc=64). * **Mamba pool sizing.** GDN layers live in a separate Mamba state pool; the `--mamba-full-memory-ratio` (defaults to 0.9) controls the split between the 10 full-attention layers' KV pool and the 30 GDN layers' conv+SSM state pool. Default split comfortably handles conc=64 on a 2×H200 node; if you need higher concurrency than `--max-running-requests` allows for your workload, raise `--mamba-full-memory-ratio` slightly (each +1% mamba ratio costs full-attn KV). * **Vision input.** Images are accepted via the standard `image_url` chat message type. Vision tokens are counted into the prompt (`prompt_tokens_details.image_tokens` shows the count), and the model honors `<|vision_start|>` / `<|vision_end|>` boundaries exactly. * **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The FP8 B200 cells inherit the verified H200 TP=1 recipe but remain unverified; treat them as a starting point until a Blackwell measurement is available. ## 3. Advanced Usage The outputs below are verbatim captures from a live server (sampling per the checkpoint's `generation_config.json`, temperature 1.0). Re-running the same request yields a semantically equivalent but textually different trace — treat them as representative, not deterministic. ### 3.1 Reasoning InternS2-Mobius is a hybrid-reasoning model — thinking traces start with "Thinking Process:" before the final answer. Enable the **`qwen3` reasoning parser** (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split thinking into `message.reasoning_content` and the answer into `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{"role": "user", "content": "What is 15% of 240?"}], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: Thinking Process: 1. **Identify the core question:** The user is asking for 15% of 240. 2. **Determine the calculation method:** To find a percentage of a number, multiply the number by the percentage expressed as a decimal or fraction. * Percentage: 15% * Decimal: 0.15 * Fraction: 15/100 3. **Perform the calculation:** $240 \times 0.15$ * Method 1: $240 \times 0.10 = 24$ (10%) and $240 \times 0.05 = 12$ (5%). Then add them: $24 + 12 = 36$. * Method 2: $240 \times 15 = 3600$. Divide by 100 -> 36. 4. **Verify the result:** The calculation is correct. 5. **Formulate the answer:** State the final number clearly.cw Answer: 15% of 240 is **36**. Here is the math: $240 \times 0.15 = 36$ ``` ### 3.2 Tool Calling Enable the **`qwen3_coder` tool-call parser** (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Intern-S2-Mobius emits `………value……` — this is exactly the format `qwen3_coder` parses; without the parser the call is left as raw text in `content`. On this thinking-mode model the turn also fills `reasoning_content`, so print both fields. **Auto-resolution works out of the box.** Intern-S2-Mobius's chat template contains the ` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, }, "required": ["location"], }, }, }] resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{"role": "user", "content": "What is the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have access to a get_weather function that can get the current weather for a location. The function requires a "location" parameter which should be the city name. In this case, the user specified "Beijing", so I should use that as the location parameter. Tool calls: [ { "id": "call_545b5956b4c3457286261490", "index": 0, "type": "function", "function": {"name": "get_weather", "arguments": "{\"location\": \"Beijing\"}"} } ] finish_reason: tool_calls ``` ### 3.3 Vision Input Intern-S2-Mobius takes images via the OpenAI-compatible `image_url` content type. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="internlm/Intern-S2-Mobius", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}}, {"type": "text", "text": "Describe this image in one sentence."}, ], }], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user wants a one-sentence description of the image. Key elements: Tiger, lying down, grass, looking at camera. Drafting: A tiger is lying in the green grass looking directly at the camera. Refining for flow and detail: A majestic tiger with striking orange and black stripes rests calmly on a bed of lush green grass, staring intently directly at the viewer. Answer: A majestic tiger with striking orange and black stripes rests calmly on a bed of lush green grass, staring intently directly at the viewer. ``` # Intern-S2-Preview Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Preview ## 1. Model Introduction **Intern-S2-Preview** is an efficient 35B scientific multimodal foundation model. Beyond conventional parameter and data scaling, Intern-S2-Preview explores task scaling: increasing the difficulty, diversity, and coverage of scientific tasks to further unlock model capabilities. **Resources:** * HuggingFace: [internLM/Intern-S2-Preview](https://huggingface.co/internLM/Intern-S2-Preview) ## 2. SGLang Installation SGLang offers multiple installation methods. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Install SGLang from source or use an NVIDIA Docker image: ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker for NVIDIA GPUs docker pull lmsysorg/sglang:latest ``` For how to actually launch a docker image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces): ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ sglang serve ``` ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware and parser configuration. ### 3.2 Configuration Tips * Use `tp>=2` for the NVIDIA deployment commands. * Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses. * Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads. * Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP. * If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, see: * [Basic API Usage](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Vision Input Intern-S2-Preview supports image inputs. Here is an example with an image: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg" }, }, { "type": "text", "text": "Describe this image in detail.", }, ], } ], max_tokens=2048, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.2 Reasoning Parser Enable streaming to read reasoning content separately from the final answer: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[ {"role": "user", "content": "Solve this step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True, ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.3 Tool Calling Serve with `--tool-call-parser qwen3_coder` enabled, then send OpenAI-compatible tool requests: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name", } }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="internLM/Intern-S2-Preview", messages=[{"role": "user", "content": "What is the weather in Beijing?"}], tools=tools, max_tokens=1024, ) print(response.choices[0].message) ``` # InternVL3.5 Source: https://docs.sglang.io/cookbook/autoregressive/InternVL/InternVL3.5 ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-internvl3-5-guide # Edit this file and submit a PR ``` ## 📚 Reference * [GLM-4.6V](../GLM/GLM-4.6V) *** **Let's build this together!** 🌟 # Jina-reranker-m0 Source: https://docs.sglang.io/cookbook/autoregressive/Jina/Jina-reranker-m0 ## 📝 Community Contribution Welcome This guide is currently under development. We welcome community contributions! If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation. ## 🚀 How to Contribute ```shell Command theme={null} git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git cd sglang-cookbook git checkout -b add-jina-reranker-m0-guide # Edit this file and submit a PR ``` ## 📚 Reference * [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md) *** **Let's build this together!** 🌟 # LFM2.5 Source: https://docs.sglang.io/cookbook/autoregressive/LiquidAI/LFM2.5 Deploy Liquid AI's LFM2.5 with SGLang — hybrid gated short conv + GQA models from 350M to the 8B-A1B MoE, plus LFM2.5-VL vision, with reasoning and Pythonic tool calling. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` LFM2.5 support — the dense / MoE / VL model classes and the `lfm2` tool-call parser — ships on SGLang `main`. If your installed release predates it, install from source or use the Docker dev image. Then run the **Python** output of the command panel below in that environment. LFM2.5 support ships in the pinned SGLang dev image: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-cu13 ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces): ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:dev-cu13 \ sglang serve ``` Every LFM2.5 model runs on a **single GPU (TP=1)** — pick your hardware + model variant to generate the launch command. One recipe covers all operating points per variant; the commands differ only by the parsers a model needs and, on Blackwell, the attention backend. The `lfm2` tool-call parser and each reasoning model's `--reasoning-parser` are already part of the verified command.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the dev image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. For LFM2.5 the exposed knob is the **TP override** (every variant is verified at TP=1; TP=2 is available for experimentation on the larger checkpoints). The reasoning and tool-call parsers are not playground toggles here — they are variant-intrinsic and already baked into each verified command. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction LFM2.5 is [Liquid AI](https://www.liquid.ai/)'s family of hybrid models for on-device deployment, released under the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). It builds on the LFM2 architecture with extended pre-training — 10T → 28T tokens for the dense models, 12T → 38T for the 8B-A1B MoE — and large-scale reinforcement learning. The backbone interleaves **gated short convolution blocks** with a small minority of **grouped query attention (GQA) blocks**. Each convolution block applies input-dependent multiplicative gating around a depthwise short convolution, giving fast local mixing at low compute and memory cost. The GQA blocks handle global context and long-range retrieval. This minimal hybrid layout was selected by a hardware-in-the-loop architecture search under edge latency and memory budgets. On CPUs it delivers up to 2× faster prefill and decode than similarly sized models (see the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404)). **Key Features:** * **Hybrid gated short conv + GQA layout**: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA). With only 6 attention layers per model, the KV cache stays small even at long context. * **Block details**: depthwise convolutions with kernel size 3; GQA with 8 KV groups and head size 64, plus RoPE and QK-Norm; pre-norm RMSNorm and SwiGLU MLPs throughout. * **Sparse MoE (8B-A1B)**: 8.3B total / 1.5B active parameters. Every layer except the first two replaces its dense MLP with a 32-expert MoE block; each token is routed to the top-4 SwiGLU experts by a normalized sigmoid router with adaptive bias load balancing. * **New in 2.5 (8B-A1B)**: the blocks are unchanged from LFM2-8B-A1B, but the context window grows from 32K to 128K (a RoPE base-θ increase plus long-context midtraining) and the vocabulary doubles from 65,536 to 128,000 tokens for more efficient non-Latin tokenization. * **Pythonic tool calling**: function calls are emitted as a Python list between `<|tool_call_start|>` and `<|tool_call_end|>` tokens. The `lfm2` tool-call parser surfaces these as standard `message.tool_calls`. * **Reasoning variants**: the 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that always emit an explicit `...` chain-of-thought before the answer. The MoE's 1.5B active parameters keep those reasoning tokens cheap. * **Multilingual**: every model except the JP checkpoints covers at least English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish (some variants add more). The dedicated JP chat checkpoints focus on Japanese (Japanese + English only). * **Vision**: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 So400M NaFlex encoder for OCR, document understanding, and multilingual vision. LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 Base-86M encoder for captioning and object detection at edge sizes; bounding-box grounding and function calling are new in the 2.5 release. **Available Models:**
Model Parameters Context Role
LFM2.5-8B-A1B 8.3B total / 1.5B active (MoE) 128K Reasoning-tuned, agentic / tool use
LFM2.5-1.2B-Instruct 1.17B (dense) 32K General instruct, RAG, data extraction
LFM2.5-1.2B-Thinking 1.17B (dense) 32K Reasoning (always-on chain-of-thought)
LFM2.5-350M 350M (dense) 32K Compact instruct, structured output
LFM2.5-230M 230M (dense) 32K Most compact; data extraction, structured output
LFM2.5-1.2B-JP-202606 1.17B (dense) 32K Japanese chat (latest)
LFM2.5-1.2B-JP 1.17B (dense) 32K Japanese chat (original)
LFM2.5-VL-1.6B 1.2B LM + SigLIP2 400M 32K Vision-language (OCR, docs, multi-image)
LFM2.5-VL-450M 350M LM + SigLIP2 86M 32K Compact vision-language (captioning, object detection)
LFM2.5-1.2B-Base 1.17B (dense) 32K Pre-trained base (no post-training)
The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (original — launch without `--tool-call-parser`) and the **Base** repos (pre-trained only, no post-training — see [§3.5](#3-5-base-checkpoints)) launch the same way with the model path swapped. **Choosing a variant:** * **8B-A1B** — flagship for agentic and tool-calling workloads; the only 128K-context option. * **1.2B-Thinking** — reasoning-heavy tasks: math, tool use, programming. * **1.2B-Instruct** — the recommended pick for chat and creative writing. * **350M** — tool use, data extraction, and structured output; not recommended for math, code, or creative writing. * **230M** — the most compact checkpoint; same use as the 350M, not for math, code, or creative writing. **License:** [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). **Resources:** [LFM2.5 announcement](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai), [LFM2.5-8B-A1B blog](https://www.liquid.ai/blog/lfm2-5-8b-a1b), [LFM docs](https://docs.liquid.ai/lfm/getting-started/welcome), [LFM2 Technical Report (arXiv:2511.23404)](https://arxiv.org/abs/2511.23404). ## 2. Configuration Tips * **Reasoning parser**: LFM2.5 reasoning models wrap their chain-of-thought in `...` tags. The command generator passes `--reasoning-parser qwen3` for **8B-A1B** (it emits an explicit opening ``) and `--reasoning-parser qwen3-thinking` for **1.2B-Thinking** (always-on reasoning). This splits the thinking process into `reasoning_content`; without it the chain-of-thought stays inline in `content`. * **Tool calling**: `--tool-call-parser lfm2` surfaces LFM2.5's Pythonic `<|tool_call_start|>[...]<|tool_call_end|>` calls as standard `message.tool_calls`. The original **1.2B-JP** does not expose tool calling; **Base** has no post-training (see [§3.5](#3-5-base-checkpoints)). * **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for the dense text models. The **8B-A1B** uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks `--attention-backend flashinfer` for it. The **VL** language model also uses that state cache and offers two backends: `--attention-backend flashinfer` (keeps prefix/radix caching — what the generator emits), or `--attention-backend trtllm_mha --disable-radix-cache` to run the language model on Blackwell `trtllm_mha` attention (`--disable-radix-cache` lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with `--mm-attention-backend fa4` for the vision tower. * **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality. * **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected. * **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs. * **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`. * **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant. **Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream.
Model temperature extra\_body (sampler)
LFM2.5-8B-A1B0.2
LFM2.5-1.2B-Instruct0.1
LFM2.5-1.2B-Thinking0.05
LFM2.5-350M0.1
LFM2.5-230M0.1
LFM2.5-1.2B-JP-2026060.1
LFM2.5-1.2B-JP0.3
LFM2.5-VL-1.6B (text)0.1
LFM2.5-VL-450M (text)0.1
LFM2.5-1.2B-Base0.3
## 3. Advanced Usage ### 3.1 Basic Usage A single client with the recommended sampling presets applied per model (the examples in the following sections reuse this `chat` helper): ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body. SAMPLING = { "LiquidAI/LFM2.5-8B-A1B": dict(temperature=0.2, extra_body={"top_k": 80, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-Instruct": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-Thinking": dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-350M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-230M": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1, extra_body={"top_k": 50, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-VL-1.6B": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}), "LiquidAI/LFM2.5-VL-450M": dict(temperature=0.1, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}), } def chat(model, messages, **overrides): cfg = SAMPLING[model] body = cfg["extra_body"] | overrides.pop("extra_body", {}) return client.chat.completions.create( model=model, messages=messages, temperature=cfg["temperature"], extra_body=body, **overrides, ) resp = chat( "LiquidAI/LFM2.5-1.2B-Instruct", [{"role": "user", "content": "What is C. elegans? Answer in one sentence."}], ) print(resp.choices[0].message.content) ``` ### 3.2 Reasoning The 8B-A1B and 1.2B-Thinking checkpoints emit chain-of-thought as a built-in behavior. The Deploy panel launches them with the matching `--reasoning-parser`, which separates the thinking process into `reasoning_content`: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-8B-A1B", [{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}], ) msg = resp.choices[0].message print("Reasoning:", msg.reasoning_content) print("Answer:", msg.content) ``` ### 3.3 Tool Calling LFM2.5 writes Pythonic tool calls. With `--tool-call-parser lfm2` (already part of the launch command) they are surfaced as standard `message.tool_calls`: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-1.2B-Instruct", [{"role": "user", "content": "What's the weather in Paris?"}], tools=[{ "type": "function", "function": { "name": "get_weather", "description": "Get current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }], ) for call in resp.choices[0].message.tool_calls or []: print(call.function.name, call.function.arguments) ``` Tool calling is supported on 8B-A1B, 1.2B-Thinking, 1.2B-Instruct, 350M, 230M, 1.2B-JP-202606, VL-1.6B, and VL-450M. For the **VL** models it is text-turn-only — do not combine an image and tools in the same turn. ### 3.4 Vision Input The VL models (VL-1.6B and VL-450M) accept images via standard OpenAI multimodal content blocks. Base64 data URIs (`data:image/jpeg;base64,...`) work in place of a URL: ```python Example theme={null} resp = chat( "LiquidAI/LFM2.5-VL-1.6B", [{ "role": "user", "content": [ {"type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}}, {"type": "text", "text": "What is in this image?"}, ], }], ) print(resp.choices[0].message.content) ``` ### 3.5 Base Checkpoints Each size ships a pre-trained Base repo — [LFM2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base), [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base), [LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base), and [LFM2.5-8B-A1B-Base](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-Base) — intended for fine-tuning and continued pre-training. The repos ship a ChatML-style chat template, so `chat.completions` requests format normally. The checkpoints have no post-training, though — don't expect instruction following. For raw text continuation: ```python Example theme={null} comp = client.completions.create( model="LiquidAI/LFM2.5-1.2B-Base", prompt="The capital of France is", temperature=0.3, extra_body={"min_p": 0.15, "repetition_penalty": 1.05}, ) print(comp.choices[0].text) ``` # LongCat-2.0 Source: https://docs.sglang.io/cookbook/autoregressive/Meituan/LongCat-2.0 Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv # Choose the nightly wheel index for your CUDA runtime. SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130 # B300 / CUDA 13 # SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129 # CUDA 12.9 uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} # Choose the rolling nightly image for your hardware. SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13 # B300 / CUDA 13 # SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev # Other supported hardware docker pull "${SGLANG_DOCKER_IMAGE}" ``` For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy: * **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill. All recipes here run the LongCat sparse-attention indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on LongCat-2.0. The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.
Model Architecture Serving precision
LongCat-2.0-FP8 Sparse MoE · LongCat Sparse Attention · n-gram embedding FP8 weights, BF16 KV cache
**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8). ## 2. Configuration Tips * **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint. * **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you. * **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path. * **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster. * **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time. * **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout. * **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe. ## 3. Advanced Usage ### 3.1 Test the deployment ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meituan-longcat/LongCat-2.0-FP8", "messages": [ {"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."} ], "max_tokens": 32, "chat_template_kwargs": {"enable_thinking": false} }' ``` ```text Output theme={null} 15 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="meituan-longcat/LongCat-2.0-FP8", messages=[ { "role": "user", "content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.", } ], max_tokens=32, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` ```text Output theme={null} 15 ``` ## 4. Validation The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.
Evaluation Examples Accuracy
GSM8K 200 98.0%
GSM8K 1314 95.8904109589041%
CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation. # Llama-3.1 Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.1 ## 1. Model Introduction Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time. These models bring open intelligence to all, with several new features and improvements: * **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities. * **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning. * **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools. * **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai. For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models. ### 3.2 Configuration Tips **Speculative Decoding (NVIDIA GPUs):** * Using Speculative Decoding for latency-sensitive scenarios: * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm * `--speculative-num-steps 3`: Number of speculative verification rounds * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens * `--speculative-num-draft-tokens 4`: Number of draft tokens per step * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B). **AMD GPU Deployment:** * **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB) * **Verified TP Configurations**: * MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1) * MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1) * **FP8 Model Variants**: * 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8` * 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV` * **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models **Xeon CPU Deployment:** * Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage SGLang exposes an OpenAI-compatible endpoint. First, start the server ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-405B-Instruct \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="Meta-Llama/Llama-3.1-405B-Instruct", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."}, ], temperature=0.2, max_tokens=512, ) print(resp.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} **Exponential Backoff Retry Function in Python** ===================================================== Below is a Python function that uses the `requests` library to retry a request with exponential backoff. ```python import requests import time import random def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60): """ Retry a request with exponential backoff. Args: url (str): The URL to make the request to. method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.). retries (int): The number of retries to attempt. Defaults to 3. backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1. max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60. Returns: The response object from the successful request. """ delay = 1 for attempt in range(retries + 1): try: response = requests.request(method, url) response.raise_for_status() # Raise an exception for HTTP errors return response except requests.RequestException as e: if attempt < retries: # Calculate the delay for this retry delay = min(delay * backoff_factor, max_delay) # Add a random jitter to the delay to prevent thundering herd problem delay += random.uniform(0, delay * 0.1) # Wait for the calculated delay before retrying time.sleep(delay) else: # If all retries have failed, raise the exception raise e ... ```` ### 4.2 Advanced Usage #### 4.2.1 Tool Calling Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled: ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-405B-Instruct \ --tool-call-parser llama3 \ --tp 8 ``` **Python Example** ```python Example theme={null} from openai import OpenAI client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "unit"], }, }, } ] response = client.chat.completions.create( model="meta-llama/Llama-3.1-405B-Instruct", messages=[ { "role": "user", "content": "What's the weather like in Boston today?", } ], temperature=0.7, stream=True, tools=tools, ) arguments = [] tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api) **Output Example** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"city": "Boston", "unit": "fahrenheit"} ``` **Handling Tool Call Results** After getting the tool call, you can execute the function: ```python Example theme={null} def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather like in Boston today?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Boston", "unit": "fahrenheit"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Boston", "fahrenheit") } ] final_response = client.chat.completions.create( model="Meta-Llama/Llama-3.1-405B-Instruct", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside" ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA A100 GPU (8x) * Model: Meta-Llama/Llama-3.1-70B * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path Meta-Llama/Llama-3.1-70B \ --tp 8 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} sglang serve \ --backend sglang \ --model Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 79.81 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4208 Request throughput (req/s): 0.13 Input token throughput (tok/s): 76.44 Output token throughput (tok/s): 52.88 Peak output token throughput (tok/s): 54.00 Peak concurrent requests: 2 Total token throughput (tok/s): 129.32 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7977.81 Median E2E Latency (ms): 6373.48 ---------------Time to First Token---------------- Mean TTFT (ms): 131.61 Median TTFT (ms): 131.77 P99 TTFT (ms): 163.88 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.63 Median TPOT (ms): 18.63 P99 TPOT (ms): 18.65 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.64 Median ITL (ms): 18.64 P95 ITL (ms): 18.69 P99 ITL (ms): 18.74 Max ITL (ms): 21.95 ================================================== ``` ##### 5.1.1.2 Medium Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 79.47 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 38450 Request throughput (req/s): 1.01 Input token throughput (tok/s): 499.17 Output token throughput (tok/s): 513.48 Peak output token throughput (tok/s): 674.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1012.65 Concurrency: 13.47 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13376.67 Median E2E Latency (ms): 14130.48 ---------------Time to First Token---------------- Mean TTFT (ms): 264.84 Median TTFT (ms): 147.02 P99 TTFT (ms): 791.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.09 Median TPOT (ms): 26.08 P99 TPOT (ms): 34.65 ---------------Inter-Token Latency---------------- Mean ITL (ms): 25.76 Median ITL (ms): 23.95 P95 ITL (ms): 24.72 P99 ITL (ms): 98.32 Max ITL (ms): 478.92 ================================================== ``` ##### 5.1.1.3 High Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 131.64 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 243641 Request throughput (req/s): 3.80 Input token throughput (tok/s): 1897.87 Output token throughput (tok/s): 1919.38 Peak output token throughput (tok/s): 3100.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3817.25 Concurrency: 89.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 23616.71 Median E2E Latency (ms): 22770.44 ---------------Time to First Token---------------- Mean TTFT (ms): 245.98 Median TTFT (ms): 184.22 P99 TTFT (ms): 1251.67 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 47.19 Median TPOT (ms): 48.67 P99 TPOT (ms): 56.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 46.34 Median ITL (ms): 33.46 P95 ITL (ms): 108.61 P99 ITL (ms): 166.11 Max ITL (ms): 1107.09 ================================================== ``` #### 5.1.2 Summarization Scenario Benchmark ##### 5.1.2.1 Low Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B\ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 83.25 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.12 Input token throughput (tok/s): 503.77 Output token throughput (tok/s): 50.69 Peak output token throughput (tok/s): 54.00 Peak concurrent requests: 2 Total token throughput (tok/s): 554.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8322.45 Median E2E Latency (ms): 6873.36 ---------------Time to First Token---------------- Mean TTFT (ms): 395.25 Median TTFT (ms): 318.02 P99 TTFT (ms): 850.80 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.80 Median TPOT (ms): 18.81 P99 TPOT (ms): 19.03 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.83 Median ITL (ms): 18.81 P95 ITL (ms): 19.06 P99 ITL (ms): 19.08 Max ITL (ms): 23.08 ================================================== ``` ##### 5.1.2.2 Medium Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 107.12 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41669 Total generated tokens (retokenized): 41603 Request throughput (req/s): 0.75 Input token throughput (tok/s): 2800.81 Output token throughput (tok/s): 389.00 Peak output token throughput (tok/s): 624.00 Peak concurrent requests: 19 Total token throughput (tok/s): 3189.81 Concurrency: 14.18 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18988.30 Median E2E Latency (ms): 20290.66 ---------------Time to First Token---------------- Mean TTFT (ms): 603.42 Median TTFT (ms): 531.82 P99 TTFT (ms): 2607.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 36.94 Median TPOT (ms): 36.73 P99 TPOT (ms): 79.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.36 Median ITL (ms): 25.72 P95 ITL (ms): 27.07 P99 ITL (ms): 439.74 Max ITL (ms): 2529.51 ================================================== ``` ##### 5.1.2.3 High Concurrency ```shell Command theme={null} sglang serve \ --backend sglang \ --model-path Meta-Llama/Llama-3.1-70B \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 215.66 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 170000 Total generated tokens (retokenized): 169035 Request throughput (req/s): 1.48 Input token throughput (tok/s): 5906.92 Output token throughput (tok/s): 788.27 Peak output token throughput (tok/s): 1920.00 Peak concurrent requests: 69 Total token throughput (tok/s): 6695.19 Concurrency: 60.01 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 40443.85 Median E2E Latency (ms): 39813.12 ---------------Time to First Token---------------- Mean TTFT (ms): 633.32 Median TTFT (ms): 616.38 P99 TTFT (ms): 1912.97 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 74.95 Median TPOT (ms): 82.85 P99 TPOT (ms): 118.46 ---------------Inter-Token Latency---------------- Mean ITL (ms): 75.08 Median ITL (ms): 34.12 P95 ITL (ms): 261.18 P99 ITL (ms): 828.12 Max ITL (ms): 1970.03 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * **Results**: ```text Output theme={null} Accuracy: 0.830 Invalid: 0.000 Latency: 11.794 s Output throughput: 1406.961 token/s ``` # Llama-3.3-70B Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.3-70B ## 1. Model Introduction [Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment. **Key Features:** * **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1 * **Tool Calling**: Native support for function calling and tool use scenarios * **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai) * **Extended Context**: 128K token context window for processing long documents and complex tasks * **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X **License:** Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details. For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models). ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs. ### 3.1 Interactive Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup. ### 3.2 Configuration Tips **AMD GPU Deployment:** * All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants * **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV` * **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support * **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput **Xeon CPU Deployment:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Tool Calling Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment: ```shell Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.3-70B-Instruct \ --tool-call-parser llama3 \ --tp 1 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=[ {"role": "user", "content": "What's the weather in Tokyo?"} ], tools=tools, temperature=0.7 ) # Check for tool calls message = response.choices[0].message if message.tool_calls: tool_call = message.tool_calls[0] print(f"Function: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") ``` **Handling Tool Call Results:** ```python Example theme={null} # After executing the function, send the result back def get_weather(location, unit="celsius"): # Your weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Build conversation with tool result messages = [ {"role": "user", "content": "What's the weather in Tokyo?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Tokyo", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Tokyo", "celsius") } ] final_response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!" ``` #### 4.2.2 Long Context Processing Leverage the 128K context window for processing long documents: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Example with long document long_document = "..." * 10000 # Your long document here response = client.chat.completions.create( model="meta-llama/Llama-3.3-70B-Instruct", messages=[ {"role": "user", "content": f"Summarize this document:\n\n{long_document}"} ], temperature=0.7, max_tokens=1000 ) print(response.choices[0].message.content) ``` ## 5. Benchmarking Use the SGLang benchmarking suite to test model performance with different workload patterns: ### 5.1 Basic Benchmark Command ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 1000 \ --random-input 1024 \ --random-output 1024 \ --max-concurrency 16 ``` ### 5.2 Adjusting Benchmark Parameters **Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns: * Short conversations: `--random-input 1024 --random-output 1024` * Long outputs: `--random-input 1024 --random-output 8192` * Long inputs: `--random-input 8192 --random-output 1024` **Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios: * Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100` * Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000` * High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000` *** ## 📚 Additional Resources * [Meta Llama Models Repository](https://github.com/meta-llama/llama-models) * [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) * [SGLang Documentation](/) * [AMD ROCm Documentation](https://rocm.docs.amd.com/) # Llama 4 Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama4 ## 1. Model Introduction [Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance. SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5). Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118). This generation delivers comprehensive upgrades across the board: The highly capable Llama 4 Maverick with 17B active parameters out of \~400B total, with 128 experts. The efficient Llama 4 Scout also has 17B active parameters out of \~109B total, using just 16 experts. Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi). For more details, please refer to the official llama4 Repository:[https://www.llama.com/models/llama-4/](https://www.llama.com/models/llama-4/) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips * **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200. * **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`: * Blackwell (B200/GB200): `trtllm_mha` * Hopper (H100/H200): `fa3` * AMD GPUs: `aiter` * Intel XPU: `intel_xpu` * Other: `triton` * **Chat Template:** Add `--chat-template llama-4` for chat completion tasks. * **Multi-Modal:** Add `--enable-multimodal` to enable image input support. * **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1). * **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Llama4 \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 4.2.2 Launch the server ### Llama-4-Scout 8-GPU deployment command: ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### Llama-4-Maverick 8-GPU deployment command: ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --tp 8 \ --trust-remote-code ``` #### 4.2.3 EAGLE Speculative Decoding SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --trust-remote-code \ --tp 8 ``` ## 5. Benchmark ### 5.1 Speed Benchmark (Scout) Test Environment: Hardware: AMD MI300x GPU Model: Llama-4-Scout Tensor Parallelism: 8 sglang version: 0.5.9 * **Model Deployment** ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 74.62 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4211 Request throughput (req/s): 0.14 Input token throughput (tok/s): 82.88 Output token throughput (tok/s): 57.42 Peak output token throughput (tok/s): 146.00 Peak concurrent requests: 2 Total token throughput (tok/s): 140.20 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7459.48 Median E2E Latency (ms): 4489.77 ---------------Time to First Token---------------- Mean TTFT (ms): 4246.98 Median TTFT (ms): 68.57 P99 TTFT (ms): 48091.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.49 Median TPOT (ms): 7.40 P99 TPOT (ms): 7.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.49 Median ITL (ms): 7.49 P95 ITL (ms): 7.47 P99 ITL (ms): 7.52 Max ITL (ms): 10.44 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 45.41 Total input tokens: 49668 Total input text tokens: 49668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40516 Request throughput (req/s): 2.26 Input token throughput (tok/s): 1120.46 Output token throughput (tok/s): 1152.47 Peak output token throughput (tok/s): 1520.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2272.84 Concurrency: 14.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6089.22 Median E2E Latency (ms): 6568.80 ---------------Time to First Token---------------- Mean TTFT (ms): 124.44 Median TTFT (ms): 87.42 P99 TTFT (ms): 268.72 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 11.88 Median TPOT (ms): 12.00 P99 TPOT (ms): 15.49 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.72 Median ITL (ms): 10.54 P95 ITL (ms): 11.22 P99 ITL (ms): 67.88 Max ITL (ms): 74.05 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Scout-17B-16E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 85.84 Total input tokens: 249841 Total input text tokens: 249841 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 250498 Request throughput (req/s): 5.84 Input token throughput (tok/s): 2910.84 Output token throughput (tok/s): 2944.82 Peak output token throughput (tok/s): 4100.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5854.65 Concurrency: 92.24 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15844.00 Median E2E Latency (ms): 15262.56 ---------------Time to First Token---------------- Mean TTFT (ms): 204.46 Median TTFT (ms): 129.96 P99 TTFT (ms): 528.54 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 41.56 Median TPOT (ms): 42.90 P99 TPOT (ms): 47.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 40.99 Median ITL (ms): 24.46 P95 ITL (ms): 84.46 P99 ITL (ms): 87.64 Max ITL (ms): 226.06 ================================================== ``` ### 5.2 Speed Benchmark (Maverick) Test Environment: Hardware: AMD MI300x GPU Model: Llama-4-Maverick Tensor Parallelism: 8 sglang version: 0.5.9 * **Model Deployment** ```bash Command theme={null} sglang serve \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --tp 8 \ --context-length 1000000 \ --trust-remote-code ``` ### 5.2.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 68.08 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4202 Request throughput (req/s): 0.15 Input token throughput (tok/s): 89.62 Output token throughput (tok/s): 61.99 Peak output token throughput (tok/s): 168.00 Peak concurrent requests: 2 Total token throughput (tok/s): 151.61 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6805.62 Median E2E Latency (ms): 2733.91 ---------------Time to First Token---------------- Mean TTFT (ms): 4296.56 Median TTFT (ms): 57.45 P99 TTFT (ms): 38633.95 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.95 Median TPOT (ms): 5.96 P99 TPOT (ms): 5.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.96 Median ITL (ms): 5.96 P95 ITL (ms): 6.02 P99 ITL (ms): 6.08 Max ITL (ms): 7.02 ================================================== ``` ### 5.2.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 30.72 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40923 Request throughput (req/s): 2.60 Input token throughput (tok/s): 1291.39 Output token throughput (tok/s): 1328.41 Peak output token throughput (tok/s): 1760.00 Peak concurrent requests: 22 Total token throughput (tok/s): 2619.80 Concurrency: 13.92 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5345.15 Median E2E Latency (ms): 5679.73 ---------------Time to First Token---------------- Mean TTFT (ms): 259.30 Median TTFT (ms): 72.60 P99 TTFT (ms): 1063.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.53 Median TPOT (ms): 10.22 P99 TPOT (ms): 20.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.99 Median ITL (ms): 9.10 P95 ITL (ms): 9.87 P99 ITL (ms): 55.62 Max ITL (ms): 868.54 ================================================== ``` ### 5.2.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 90.95 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 251625 Request throughput (req/s): 5.50 Input token throughput (tok/s): 2746.77 Output token throughput (tok/s): 2777.90 Peak output token throughput (tok/s): 3700.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5524.67 Concurrency: 93.04 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16924.17 Median E2E Latency (ms): 16294.85 ---------------Time to First Token---------------- Mean TTFT (ms): 188.19 Median TTFT (ms): 128.96 P99 TTFT (ms): 534.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.63 Median TPOT (ms): 35.37 P99 TPOT (ms): 38.26 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.19 Median ITL (ms): 27.66 P95 ITL (ms): 76.91 P99 ITL (ms): 78.82 Max ITL (ms): 268.17 ================================================== ``` ### 5.3 Accuracy Benchmark #### 5.3.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * Llama-4-Scout-17B-16E-Instruct ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 12.731 s Output throughput: 1595.418 token/s ``` * Llama-4-Maverick-17B-128E-Instruct ```text Output theme={null} Accuracy: 0.895 Invalid: 0.000 Latency: 9.739 s Output throughput: 2405.505 token/s ``` #### 5.3.2 MMLU Pro with lm-eval Accuracy on MMLU Pro matches [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) on 8×H100 (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)):
Model Official SGLang
Llama-4-Scout-17B-16E-Instruct 74.3 75.2
Llama-4-Maverick-17B-128E-Instruct 80.5 80.7
**Scout:** ```bash Command theme={null} # Start the server python -m sglang.launch_server \ --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \ --port 30000 \ --tp 8 \ --mem-fraction-static 0.8 \ --context-length 65536 # Run lm_eval lm_eval --model local-chat-completions \ --model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \ --tasks mmlu_pro \ --batch_size 128 \ --apply_chat_template \ --num_fewshot 0 ``` **Maverick:** ```bash Command theme={null} # Start the server python -m sglang.launch_server \ --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \ --port 30000 \ --tp 8 \ --mem-fraction-static 0.8 \ --context-length 65536 # Run lm_eval lm_eval --model local-chat-completions \ --model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \ --tasks mmlu_pro \ --batch_size 128 \ --apply_chat_template \ --num_fewshot 0 ``` # Muse Glimmer Source: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseGlimmer A multimodal reasoning model served from a BF16, NVFP4 + MXFP8, vendor GGUF, or MLX checkpoint. ## Deployment See the [official SGLang installation guide](../../../docs/get-started/install) for all installation methods and hardware platforms. The steps below match the **Python** and **Docker** options in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv # Muse Glimmer support is not in a release yet -- build the PR branch: # https://github.com/sgl-project/sglang/pull/34262 git clone -b muse-glimmer https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e "python[all]" ``` Run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-muse-glimmer ``` See [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker) to start the image. Replace the inner `sglang serve ...` command with the command from the panel below. Select a checkpoint format. Select whether to use speculative decoding: * **Standard**: Use normal autoregressive decoding. * **DFlash**: Use speculative decoding with the DFlash draft model. The draft serves as published, with no conversion step. See [§2](#2-configuration-tips). ## Playground Use the Playground to test SGLang features that are not in the verified matrix. The Deploy panel above shows only combinations that the SGLang team has verified. The Playground lets you add more options to the command from the Deploy panel. ## 1. Model Introduction Muse Glimmer is a multimodal reasoning model. You can serve Muse Glimmer in four formats: * A BF16 checkpoint (`MuseGlimmerForConditionalGeneration`). * A set of vendor GGUF files. * A ready-to-serve NVFP4 + MXFP8 checkpoint. * Three MLX repacks for Apple Silicon.
Form Source Notes
BF16 meta-models/Muse-Glimmer-30B Supports image input.
GGUF Q4\_K\_M meta-models/Muse-Glimmer-30B-GGUF Text only. This path is not optimized. SGLang shows a warning at startup.
NVFP4 RadixArk/Muse-Glimmer-NVFP4 Text only. Ready to serve, no conversion needed.
MLX Q4 RadixArk/Muse-Glimmer-q4-MLX Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
MLX Q4\_K\_M (gs128) RadixArk/Muse-Glimmer-q4km-gs128-MLX Text only. Apple Silicon (MLX backend). Carries the vendor GGUF's exact quantization codes in MLX format. The measured MLX artifact. See §3.4.
MLX Q4\_K (dynamic) RadixArk/Muse-Glimmer-q4k-dynamic-MLX Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.
**Resources:** [Muse-Glimmer-30B (BF16)](https://huggingface.co/meta-models/Muse-Glimmer-30B) · [Muse-Glimmer-30B-assistant (DFlash draft)](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) · [Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) · [Muse-Glimmer-NVFP4](https://huggingface.co/RadixArk/Muse-Glimmer-NVFP4) · MLX · [q4](https://huggingface.co/RadixArk/Muse-Glimmer-q4-MLX) · [q4km-gs128](https://huggingface.co/RadixArk/Muse-Glimmer-q4km-gs128-MLX) · [q4k-dynamic](https://huggingface.co/RadixArk/Muse-Glimmer-q4k-dynamic-MLX). ## 2. Configuration Tips **The GGUF format is text only.** SGLang has no `mmproj` path. You cannot use the vision GGUF files. Use the BF16 checkpoint for multimodal input. **The NVFP4 checkpoint.** `RadixArk/Muse-Glimmer-NVFP4` is a ready-to-serve NVFP4 + MXFP8 checkpoint. No conversion needed — point `--model-path` straight at it. **The DFlash draft.** `meta-models/Muse-Glimmer-30B-assistant` is the vendor's native draft export and serves directly. No conversion needed. **DFlash with a GGUF target model** needs `--speculative-draft-load-format auto`. Without this flag, the draft model uses the `gguf` load format from the target model. The loader then rejects the draft directory. **Apple Silicon uses an MLX checkpoint, not the GGUF files.** The MLX backend has no GGUF path. Serve one of the three `RadixArk/Muse-Glimmer-*-MLX` artifacts with `SGLANG_USE_MLX=1` (see the Apple Silicon cells in the command panel). All three take the same flags; `q4km-gs128` is the one with a measured round. Keep `--disable-radix-cache` — the windowed KV storage for the sliding-window layers requires it — and set `SGLANG_MLX_CACHE_LIMIT_GB=8` so the MLX buffer cache does not grow the footprint under concurrent load. Speculative decoding is not available on the MLX backend. ## 3. Advanced Usage ### 3.1 Reasoning Muse Glimmer enables the `muse` reasoning parser by default. This parser separates the reasoning text from the final answer. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="meta-models/Muse-Glimmer-30B", messages=[{"role": "user", "content": "What is 15% of 240?"}], ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ### 3.2 Tool Calling Muse Glimmer enables the `muse` tool-call parser by default. This parser sends structured tool calls in `message.tool_calls`. ### 3.3 Multimodal The BF16 checkpoint supports image input. It defaults to text only. To switch, select **Modality** in the command panel above. **Text only** adds `--language-model-only`. This flag turns off the vision tower. SGLang does not build or load the vision weights. This frees memory for the KV cache. SGLang rejects image requests in this mode. Select **Image + text** to turn on image input. NVFP4, GGUF, and the MLX artifacts are text only. The Modality option does not appear for GGUF or MLX; NVFP4 only offers **Text only**. ### 3.4 Apple Silicon (MLX) The MLX backend serves three Muse Glimmer artifacts on Apple Silicon Macs (48 GB unified memory or more). All three are text only — the MLX backend has no vision path — and all three take the same flags, so pick one in the command panel: * `RadixArk/Muse-Glimmer-q4-MLX` — no measured round yet. * `RadixArk/Muse-Glimmer-q4km-gs128-MLX` — a lossless repack of the vendor's Q4\_K\_M (gs128) GGUF: every weight keeps the GGUF's exact quantization code, with the group scales re-expressed in MLX affine bf16 (≤2⁻⁸ relative rounding). The numbers below are for this artifact. * `RadixArk/Muse-Glimmer-q4k-dynamic-MLX` — no measured round yet. Choose along the speed-versus-accuracy axis: footprint and expected accuracy both grow `q4` → `q4km-gs128` → `q4k-dynamic`, and decode speed moves the other way. Decode on Apple Silicon is memory-bandwidth-bound, so a smaller artifact reads fewer weight bytes per token — more tokens per second, and more unified memory left over for the KV cache. Take `q4` for the fastest responses on the smallest machine, `q4k-dynamic` to stay closest to BF16, and `q4km-gs128` for the middle ground — it is also the only one of the three with a measured round, below. This table shows accuracy for the gs128 checkpoint, with the vendor llama.cpp fork serving the source GGUF on the same machine as the reference. GSM8K: 200 questions, no-thinking chat template, temperature 0, max 2048 new tokens. CIMemories: 1 profile, full combo, single trial, DeepSeek-R1-0528 judge.
Benchmark SGLang MLX llama.cpp (same GGUF)
GSM8K (200q, no-thinking, greedy) 0.970 0.970
CIMemories — violation rate (lower is better) 0.00% 8.27%
CIMemories — coverage (higher is better) 76.0% 68.4%
CIMemories is a single-trial benchmark with a nondeterministic judge; treat the SGLang-vs-llama.cpp gap on that row as run noise, not a runtime effect. GSM8K parity is exact. Decode throughput for gs128 on an M5 Pro (64 GB), 1k-in/1k-out greedy: 15.3 tok/s at batch 1, rising to 52.6 tok/s aggregate at batch 8 — ahead of llama.cpp on the same GGUF codes at every batch size above 1. # MiniMax-M2 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2 ## 1. Model Introduction [MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence. This generation delivers comprehensive upgrades across the board: * **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally. * **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages. * **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps. * **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks. For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install. ### 2.1 AMD Docker #### 2.1.1 Launch docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Minimax \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 2.1.2 Make modifications inside the docker ```shell Command theme={null} mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \ /sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py ``` #### 2.1.3 Fix torch compile Comment out the following line: @torch.compile(dynamic=True, backend=get\_compiler\_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax\_m2.py ```shell Command theme={null} #@torch.compile(dynamic=True, backend=get_compiler_backend()) ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. #### 3.1.1 NVIDIA GPU Deployment The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands: **4-GPU deployment (up to 400K context):** ```bash Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --host 0.0.0.0 \ --trust-remote-code \ --port 30000 \ --mem-fraction-static 0.85 ``` **8-GPU deployment (up to 3M context):** ```bash Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 8 \ --ep-size 8 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --host 0.0.0.0 \ --trust-remote-code \ --port 30000 \ --mem-fraction-static 0.85 ``` ### 3.2 System Requirements Recommended configurations — actual requirements depend on workload:
GPUs Context Length Support
4× 96 GB GPUs Up to 400K tokens
8× 144 GB GPUs Up to 3M tokens
### 3.3 Testing Deployment After the server starts, verify with: ```shell Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Server Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Test Code: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.6, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` Output Example: ```text Output theme={null} First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction. The problem is: What is 15% of 240? To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15. Therefore, 15% of 240 is 240 times 0.15. Let me calculate that: 240 × 0.15. I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100. Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600. Then, divide by 100: 3600 / 100 = 36. So, 15% of 240 is 36. I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer. Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36. All methods confirm it's 36. The user said "solve this problem step by step," so I should present the steps clearly. Step 1: Understand that "15% of 240" means 15 per hundred of 240. Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15. Step 3: Multiply the number by the decimal: 240 × 0.15. Step 4: Calculate the multiplication: 240 × 0.15 = 36. Step 5: Therefore, 15% of 240 is 36. I should also mention that percentage means per hundred, so it's straightforward. Finally, I need to box the answer as per the instruction. So, the final answer is 36. To find 15% of 240, follow these steps: 1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100. 2. **Convert to a decimal**: 15/100 = 0.15. 3. **Multiply by the number**: 240 × 0.15. 4. **Calculate the result**: - 240 × 0.15 = 36. Alternatively, you can break it down: - 10% of 240 is 24 (since 240 ÷ 10 = 24). - 5% of 240 is half of 10%, which is 12. - Therefore, 15% is 10% + 5% = 24 + 12 = 36. Both methods confirm the result. **Answer**: 36 ``` ### 4.2.2 Tool Calling Server Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --tool-call-parser minimax-m2 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Test Code: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` Output Example: ```text Output theme={null} Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me. Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me. Looking at the tool's parameters, I can see it requires: 1. location - which is required and should be a string representing the city name 2. unit - which is optional and can be either "celsius" or "fahrenheit" For the location parameter, I'll use "Beijing" since that's what the user asked about. For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information. So I need to make a tool call to get_weather with the following parameters: - location: "Beijing" - unit: "celsius" This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions. 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment**: * Hardware: AMD MI300X GPU(4x) * Model: MiniMax-M2 * Tensor Parallelism: 4 * sglang version: 0.5.7 **Model Deployment**: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 138.91 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.07 Input token throughput (tok/s): 43.92 Output token throughput (tok/s): 30.38 Peak output token throughput (tok/s): 46.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.30 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13887.62 Median E2E Latency (ms): 10377.26 ---------------Time to First Token---------------- Mean TTFT (ms): 4528.94 Median TTFT (ms): 385.23 P99 TTFT (ms): 38338.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 22.21 Median TPOT (ms): 22.24 P99 TPOT (ms): 22.25 ---------------Inter-Token Latency---------------- Mean ITL (ms): 22.23 Median ITL (ms): 22.24 P95 ITL (ms): 22.35 P99 ITL (ms): 22.41 Max ITL (ms): 23.64 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 81.07 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40803 Request throughput (req/s): 0.99 Input token throughput (tok/s): 489.29 Output token throughput (tok/s): 503.32 Peak output token throughput (tok/s): 704.00 Peak concurrent requests: 19 Total token throughput (tok/s): 992.61 Concurrency: 13.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13925.95 Median E2E Latency (ms): 14348.75 ---------------Time to First Token---------------- Mean TTFT (ms): 532.32 Median TTFT (ms): 147.69 P99 TTFT (ms): 1978.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 27.49 Median TPOT (ms): 26.56 P99 TPOT (ms): 46.52 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.31 Median ITL (ms): 23.47 P95 ITL (ms): 24.37 P99 ITL (ms): 125.10 Max ITL (ms): 1192.51 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 153.71 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 250982 Request throughput (req/s): 3.25 Input token throughput (tok/s): 1625.33 Output token throughput (tok/s): 1643.75 Peak output token throughput (tok/s): 2597.00 Peak concurrent requests: 107 Total token throughput (tok/s): 3269.09 Concurrency: 91.14 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 28017.24 Median E2E Latency (ms): 26865.28 ---------------Time to First Token---------------- Mean TTFT (ms): 387.41 Median TTFT (ms): 183.90 P99 TTFT (ms): 1192.44 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 55.23 Median TPOT (ms): 57.84 P99 TPOT (ms): 70.23 ---------------Inter-Token Latency---------------- Mean ITL (ms): 54.79 Median ITL (ms): 39.01 P95 ITL (ms): 143.10 P99 ITL (ms): 150.46 Max ITL (ms): 986.14 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Server Command**: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2 \ --tp-size 4 \ --trust-remote-code \ --mem-fraction-static 0.85 ``` * **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * **Result**: * MiniMax-M2 ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 15.120 s Output throughput: 1306.711 token/s ``` # MiniMax-M2.5 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.5 ## 1. Model Introduction [MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is a powerful language model developed by MiniMax, built for real-world productivity with state-of-the-art performance across coding, reasoning, agentic tasks, and tool use. As the latest iteration in the MiniMax model series, MiniMax-M2.5 achieves comprehensive enhancements across multiple domains. Details are as follows: * **Superior coding performance**: Achieves 79.7 on Droid and 76.1 on OpenCode, surpassing Opus 4.6 (78.9 and 75.9 respectively). Strong results on SWE-bench Verified, SWE-bench Multilingual, SWE-bench-pro, and Multi-SWE-bench. * **Advanced reasoning**: Demonstrates strong performance on AIME25 and other reasoning benchmarks, with robust tool use during inference. * **More capable agents**: Excels in agentic tasks including web browsing (BrowseComp, Wide Search), information retrieval (RISE), and complex tool use scenarios (Terminal Bench 2, MEWC, Finance Modeling). * **Real-world productivity**: Designed for production-grade workloads with strong performance on practical coding, data analysis, and multi-step reasoning tasks. For more details, please refer to the [official MiniMax-M2.5 announcement](https://www.minimax.io/news/minimax-m25). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **For AMD MI300X/MI325X/MI355X GPUs:** ```bash Command theme={null} # Docker (AMD MI300X/MI325X) docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x # Docker (AMD MI355X) docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x ``` ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function calling support `minimax-m2`
`--reasoning-parser` Reasoning parser for thinking mode `minimax-append-think`
`--trust-remote-code` Required for MiniMax model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.85`
`--tp` Tensor parallelism size `2` (2-GPU) or `4` (4-GPU) or `8` (8-GPU)
`--ep` Expert parallelism size `8` (NVIDIA 8-GPU) or EP=TP (AMD)
`--kv-cache-dtype` KV cache data type (AMD only) `fp8_e4m3`
`--attention-backend` Attention backend (AMD only) `triton`
**Hardware Requirements: NVIDIA** * **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8 **Hardware Requirements: AMD** * **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2 * **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8 ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2.5", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Example Output**: ```text Output theme={null} The user asks: "Who won the world series in 2020?" That is a straightforward factual question. The answer: the Los Angeles Dodgers. They won the 2020 World Series, beating the Tampa Bay Rays. The user is presumably expecting that answer. We must follow the policies. The question is safe: no disallowed content. It's just a factual question. Provide answer. We must ensure compliance: Use no disallowed content. Should we provide context? Just answer straightforwardly. The user simply asks "Who won the world series in 2020?" We'll answer: The Los Angeles Dodgers. No additional relevant info needed, but could elaborate briefly: They beat the Tampa Bay Rays in six games, the series was played in a bubble at Globe Life Field in Arlington, Texas due to COVID-19. No need for any extra. That's it. The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games. ``` ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser MiniMax-M2.5 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Streaming with Thinking Process** With `minimax-append-think`, the thinking content is wrapped in `...` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream, separating ... from content in_think = False think_printed_header = False content_printed_header = False buffer = "" for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: buffer += delta.content while buffer: if in_think: # Look for closing tag end_idx = buffer.find("") if end_idx != -1: print(buffer[:end_idx], end="", flush=True) buffer = buffer[end_idx + len(""):] in_think = False else: # Still in thinking, print what we have print(buffer, end="", flush=True) buffer = "" else: # Look for opening tag start_idx = buffer.find("") if start_idx != -1: # Print any content before before = buffer[:start_idx] if before: if not content_printed_header: print("=============== Content =================", flush=True) content_printed_header = True print(before, end="", flush=True) buffer = buffer[start_idx + len(""):] in_think = True if not think_printed_header: print("=============== Thinking =================", flush=True) think_printed_header = True else: # No tag, print as content if not content_printed_header and think_printed_header: print("\n=============== Content =================", flush=True) content_printed_header = True print(buffer, end="", flush=True) buffer = "" print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" This is straightforward: 15% = 0.15; 0.15*240 = 36. So answer: 36. Provide step-by-step: convert percent to decimal, multiply. We need to obey policies. There's no policy violation. Just answer. Provide step by step. Should respond with solution. We can also mention alternative method: 15% = 15/100 = 3/20. Multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36. Thus answer 36. We can add step-by-step. That's it. =============== Content ================= **Step‑by‑step solution** 1. **Convert the percent to a decimal** \[ 15\% = \frac{15}{100}=0.15 \] 2. **Multiply the decimal by the number** \[ 0.15 \times 240 = 36 \] (You can also think of it as \(15\% = \frac{3}{20}\) and then \(240 \times \frac{3}{20}=12 \times 3 = 36\).) \[ \boxed{36} \] ``` **Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `...` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately. #### 4.2.2 Tool Calling MiniMax-M2.5 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Non-streaming request response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7 ) message = response.choices[0].message # Check for tool calls if message.tool_calls: for tool_call in message.tool_calls: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") else: print(message.content) ``` **Output Example**: ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Note:** * Tool calls are returned in `message.tool_calls` with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.5", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment**: * Hardware: NVIDIA B200 GPU (8x) * Model: MiniMax-M2.5 * Tensor Parallelism: 8 * Expert Parallelism: 8 * sglang version: 0.5.8 #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 42.99 Total input tokens: 6091 Total input text tokens: 6091 Total generated tokens: 4220 Total generated tokens (retokenized): 3804 Request throughput (req/s): 0.23 Input token throughput (tok/s): 141.70 Output token throughput (tok/s): 98.17 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 239.87 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4295.92 Median E2E Latency (ms): 3419.28 P90 E2E Latency (ms): 7832.04 P99 E2E Latency (ms): 9601.40 ---------------Time to First Token---------------- Mean TTFT (ms): 130.57 Median TTFT (ms): 116.10 P99 TTFT (ms): 190.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.89 Median TPOT (ms): 9.89 P99 TPOT (ms): 9.91 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.89 Median ITL (ms): 9.89 P95 ITL (ms): 10.15 P99 ITL (ms): 10.32 Max ITL (ms): 14.46 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 48.43 Total input tokens: 39588 Total input text tokens: 39588 Total generated tokens: 40805 Total generated tokens (retokenized): 37142 Request throughput (req/s): 1.65 Input token throughput (tok/s): 817.37 Output token throughput (tok/s): 842.49 Peak output token throughput (tok/s): 1184.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1659.86 Concurrency: 13.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8274.32 Median E2E Latency (ms): 8692.90 P90 E2E Latency (ms): 13690.70 P99 E2E Latency (ms): 16104.18 ---------------Time to First Token---------------- Mean TTFT (ms): 305.44 Median TTFT (ms): 106.75 P99 TTFT (ms): 1053.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.20 Median TPOT (ms): 16.06 P99 TPOT (ms): 26.75 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.65 Median ITL (ms): 13.63 P95 ITL (ms): 14.90 P99 ITL (ms): 87.99 Max ITL (ms): 483.53 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 92.31 Total input tokens: 249331 Total input text tokens: 249331 Total generated tokens: 252662 Total generated tokens (retokenized): 218975 Request throughput (req/s): 5.42 Input token throughput (tok/s): 2700.94 Output token throughput (tok/s): 2737.02 Peak output token throughput (tok/s): 4479.00 Peak concurrent requests: 109 Total token throughput (tok/s): 5437.97 Concurrency: 91.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16835.82 Median E2E Latency (ms): 16042.08 P90 E2E Latency (ms): 31027.63 P99 E2E Latency (ms): 34787.91 ---------------Time to First Token---------------- Mean TTFT (ms): 391.06 Median TTFT (ms): 133.12 P99 TTFT (ms): 1712.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.04 Median TPOT (ms): 34.29 P99 TPOT (ms): 41.98 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.61 Median ITL (ms): 21.67 P95 ITL (ms): 87.76 P99 ITL (ms): 118.81 Max ITL (ms): 1145.62 ================================================== ``` #### 5.1.2 Summarization Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.49 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.23 Input token throughput (tok/s): 964.42 Output token throughput (tok/s): 97.04 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1061.46 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4346.83 Median E2E Latency (ms): 3508.84 P90 E2E Latency (ms): 7972.23 P99 E2E Latency (ms): 9659.71 ---------------Time to First Token---------------- Mean TTFT (ms): 131.50 Median TTFT (ms): 126.76 P99 TTFT (ms): 182.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.00 Median TPOT (ms): 10.01 P99 TPOT (ms): 10.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.01 Median ITL (ms): 10.02 P95 ITL (ms): 10.29 P99 ITL (ms): 10.44 Max ITL (ms): 14.11 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 50.12 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41662 Request throughput (req/s): 1.60 Input token throughput (tok/s): 5986.00 Output token throughput (tok/s): 831.38 Peak output token throughput (tok/s): 1152.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6817.38 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8727.66 Median E2E Latency (ms): 9170.52 P90 E2E Latency (ms): 14220.00 P99 E2E Latency (ms): 16896.54 ---------------Time to First Token---------------- Mean TTFT (ms): 282.56 Median TTFT (ms): 149.37 P99 TTFT (ms): 1278.62 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.60 Median TPOT (ms): 16.61 P99 TPOT (ms): 25.17 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.24 Median ITL (ms): 13.89 P95 ITL (ms): 15.96 P99 ITL (ms): 105.79 Max ITL (ms): 1065.02 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 93.92 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169999 Request throughput (req/s): 3.41 Input token throughput (tok/s): 13563.30 Output token throughput (tok/s): 1810.01 Peak output token throughput (tok/s): 2881.00 Peak concurrent requests: 71 Total token throughput (tok/s): 15373.31 Concurrency: 58.87 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 17277.69 Median E2E Latency (ms): 16827.33 P90 E2E Latency (ms): 29045.40 P99 E2E Latency (ms): 33496.77 ---------------Time to First Token---------------- Mean TTFT (ms): 692.26 Median TTFT (ms): 188.46 P99 TTFT (ms): 4932.70 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.19 Median TPOT (ms): 32.69 P99 TPOT (ms): 50.46 ---------------Inter-Token Latency---------------- Mean ITL (ms): 31.28 Median ITL (ms): 21.59 P95 ITL (ms): 101.35 P99 ITL (ms): 136.74 Max ITL (ms): 4649.23 ================================================== ``` #### 5.1.3 H100 Benchmark **Test Environment**: * Hardware: NVIDIA H100 80GB HBM3 GPU (8x) * Model: MiniMax-M2.5 * Tensor Parallelism: 8 * Expert Parallelism: 8 * sglang version: 0.5.9 * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.5 \ --tp 8 \ --ep 8 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 \ --tool-call-parser minimax-m2 ``` ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.44 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.28 Input token throughput (tok/s): 172.16 Output token throughput (tok/s): 119.08 Peak output token throughput (tok/s): 127.00 Peak concurrent requests: 2 Total token throughput (tok/s): 291.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3542.38 Median E2E Latency (ms): 2791.92 P90 E2E Latency (ms): 6317.77 P99 E2E Latency (ms): 7780.15 ---------------Time to First Token---------------- Mean TTFT (ms): 145.20 Median TTFT (ms): 80.38 P99 TTFT (ms): 633.08 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.05 Median TPOT (ms): 8.08 P99 TPOT (ms): 8.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.07 Median ITL (ms): 8.08 P95 ITL (ms): 8.12 P99 ITL (ms): 8.16 Max ITL (ms): 10.10 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 43.68 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40805 Request throughput (req/s): 1.83 Input token throughput (tok/s): 908.19 Output token throughput (tok/s): 934.22 Peak output token throughput (tok/s): 1184.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1842.42 Concurrency: 13.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7551.91 Median E2E Latency (ms): 8094.28 P90 E2E Latency (ms): 12606.99 P99 E2E Latency (ms): 14977.84 ---------------Time to First Token---------------- Mean TTFT (ms): 116.86 Median TTFT (ms): 82.33 P99 TTFT (ms): 240.59 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.81 Median TPOT (ms): 14.98 P99 TPOT (ms): 17.98 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.61 Median ITL (ms): 13.50 P95 ITL (ms): 14.15 P99 ITL (ms): 66.52 Max ITL (ms): 107.39 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 80.63 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252331 Request throughput (req/s): 6.20 Input token throughput (tok/s): 3098.45 Output token throughput (tok/s): 3133.56 Peak output token throughput (tok/s): 4800.00 Peak concurrent requests: 113 Total token throughput (tok/s): 6232.01 Concurrency: 90.56 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14604.59 Median E2E Latency (ms): 14044.04 P90 E2E Latency (ms): 26456.53 P99 E2E Latency (ms): 30136.68 ---------------Time to First Token---------------- Mean TTFT (ms): 149.32 Median TTFT (ms): 95.16 P99 TTFT (ms): 374.62 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 28.92 Median TPOT (ms): 30.09 P99 TPOT (ms): 34.31 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.66 Median ITL (ms): 21.52 P95 ITL (ms): 66.90 P99 ITL (ms): 96.76 Max ITL (ms): 376.34 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Benchmark Command: ```shell Command theme={null} python benchmark/gsm8k/bench_sglang.py --port 30000 ``` * Test Results: ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 18.033 s Output throughput: 1130.161 token/s ``` #### 5.2.2 MMLU Benchmark * Benchmark Command: ```shell Command theme={null} cd benchmark/mmlu bash download_data.sh python3 bench_sglang.py --port 30000 ``` * Test Results: ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.620 subject: anatomy, #q:135, acc: 0.830 subject: astronomy, #q:152, acc: 0.928 subject: business_ethics, #q:100, acc: 0.810 subject: clinical_knowledge, #q:265, acc: 0.891 subject: college_biology, #q:144, acc: 0.951 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.820 subject: college_mathematics, #q:100, acc: 0.660 subject: college_medicine, #q:173, acc: 0.832 subject: college_physics, #q:102, acc: 0.814 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.915 subject: econometrics, #q:114, acc: 0.719 subject: electrical_engineering, #q:145, acc: 0.834 subject: elementary_mathematics, #q:378, acc: 0.902 subject: formal_logic, #q:126, acc: 0.698 subject: global_facts, #q:100, acc: 0.710 subject: high_school_biology, #q:310, acc: 0.926 subject: high_school_chemistry, #q:203, acc: 0.793 subject: high_school_computer_science, #q:100, acc: 0.910 subject: high_school_european_history, #q:165, acc: 0.879 subject: high_school_geography, #q:198, acc: 0.955 subject: high_school_government_and_politics, #q:193, acc: 0.964 subject: high_school_macroeconomics, #q:390, acc: 0.908 subject: high_school_mathematics, #q:270, acc: 0.600 subject: high_school_microeconomics, #q:238, acc: 0.954 subject: high_school_physics, #q:151, acc: 0.781 subject: high_school_psychology, #q:545, acc: 0.956 subject: high_school_statistics, #q:216, acc: 0.847 subject: high_school_us_history, #q:204, acc: 0.922 subject: high_school_world_history, #q:237, acc: 0.916 subject: human_aging, #q:223, acc: 0.839 subject: human_sexuality, #q:131, acc: 0.893 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.861 subject: logical_fallacies, #q:163, acc: 0.890 subject: machine_learning, #q:112, acc: 0.750 subject: management, #q:103, acc: 0.883 subject: marketing, #q:234, acc: 0.944 subject: medical_genetics, #q:100, acc: 0.920 subject: miscellaneous, #q:783, acc: 0.936 subject: moral_disputes, #q:346, acc: 0.829 subject: moral_scenarios, #q:895, acc: 0.632 subject: nutrition, #q:306, acc: 0.863 subject: philosophy, #q:311, acc: 0.833 subject: prehistory, #q:324, acc: 0.907 subject: professional_accounting, #q:282, acc: 0.720 subject: professional_law, #q:1534, acc: 0.640 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.871 subject: public_relations, #q:110, acc: 0.773 subject: security_studies, #q:245, acc: 0.845 subject: sociology, #q:201, acc: 0.930 subject: us_foreign_policy, #q:100, acc: 0.940 subject: virology, #q:166, acc: 0.614 subject: world_religions, #q:171, acc: 0.895 Total latency: 81.468 Average accuracy: 0.825 ``` # MiniMax-M2.7 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.7 ## 1. Model Introduction [MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search. Key highlights: * **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**. * **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems. * **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier). * **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making. For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en). **License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA A100 / H100 / H200 / B200 `lmsysorg/sglang:v0.5.10.post1`
NVIDIA B300 / GB300 `lmsysorg/sglang:v0.5.10.post1-cu130`
AMD MI300X / MI325X `lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`
AMD MI355X `lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`
## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function calling support `minimax-m2`
`--reasoning-parser` Reasoning parser for thinking mode `minimax-append-think`
`--trust-remote-code` Required for MiniMax model loading Always enabled
`--mem-fraction-static` Static memory fraction for KV cache `0.85`
`--tp` Tensor parallelism size `2` / `4` / `8` depending on hardware
`--ep` Expert parallelism size `8` (NVIDIA 8-GPU) or EP=TP (AMD)
`--kv-cache-dtype` KV cache data type (AMD only) `fp8_e4m3`
`--attention-backend` Attention backend (AMD only) `triton`
**Hardware Requirements: NVIDIA** * **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8 **Hardware Requirements: NVIDIA GB300** * **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2 * **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput **Hardware Requirements: AMD** * **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2 * **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4 * **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8 **Hardware Requirements: Intel Xeon CPU** * It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server. * For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Deployment Command:** ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-M2.7", "messages": [ {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]}, {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print(response.choices[0].message.content) ``` **Example Output**: ```text Output theme={null} The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly. We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context. We should answer concisely. Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas. We must avoid disallowed content, no issue. Thus final. The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic. ``` ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Streaming with Thinking Process** With `minimax-append-think`, the thinking content is wrapped in `...` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream, separating ... from content in_think = False think_printed_header = False content_printed_header = False buffer = "" for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: buffer += delta.content while buffer: if in_think: # Look for closing tag end_idx = buffer.find("") if end_idx != -1: print(buffer[:end_idx], end="", flush=True) buffer = buffer[end_idx + len(""):] in_think = False else: # Still in thinking, print what we have print(buffer, end="", flush=True) buffer = "" else: # Look for opening tag start_idx = buffer.find("") if start_idx != -1: # Print any content before before = buffer[:start_idx] if before: if not content_printed_header: print("=============== Content =================", flush=True) content_printed_header = True print(before, end="", flush=True) buffer = buffer[start_idx + len(""):] in_think = True if not think_printed_header: print("=============== Thinking =================", flush=True) think_printed_header = True else: # No tag, print as content if not content_printed_header and think_printed_header: print("\n=============== Content =================", flush=True) content_printed_header = True print(buffer, end="", flush=True) buffer = "" print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation. But also ensure we follow any policy? No issues. Just straightforward. I'll provide a step-by-step solution. Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36. Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36. All good. =============== Content ================= **Step‑by‑step solution** 1. **Convert the percent to a decimal (or a fraction).** 15% = 15/100 = 0.15 = 3/20 2. **Multiply the original number (240) by this decimal/fraction.** Using the decimal: 240 × 0.15 = 36 Or using the fraction: 240 × 3/20 = (240/20) × 3 = 12 × 3 = 36 3. **Result:** 15% of 240 = **36** *Check:* - 10% of 240 = 24 - 5% of 240 = 12 - Adding them: 24 + 12 = 36, which matches the calculation. ``` **Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `...` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately. #### 4.2.2 Tool Calling MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M2.7 \ --tp 4 \ --tool-call-parser minimax-m2 \ --reasoning-parser minimax-append-think \ --trust-remote-code \ --mem-fraction-static 0.85 ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Non-streaming request response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools ) message = response.choices[0].message # Check for tool calls if message.tool_calls: for tool_call in message.tool_calls: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") else: print(message.content) ``` **Output Example**: ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M2.7", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently 22°C and sunny. ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. **Test Environment**: * Hardware: 2× NVIDIA GB300 (275GB per die) * Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130` * Model: MiniMax-M2.7 (FP8) * Tensor Parallelism: 2 * SGLang version: 0.5.10.post1 ### 5.1 Accuracy Benchmark **Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills) **Evaluation Settings**: temperature=0.6, top\_p=0.95, 8 seeds, max\_tokens=120,000, `parse_reasoning=True` #### 5.1.1 GPQA Diamond * Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions) * Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) * Evaluation command: ```bash Command theme={null} ns prepare_data gpqa ns eval \ --cluster=local \ --server_type=openai \ --model=MiniMaxAI/MiniMax-M2.7 \ --server_address=http://localhost:30000/v1 \ --output_dir=./m2.7-eval/ \ --benchmarks=gpqa:8 \ ++prompt_config=eval/aai/mcq-4choices \ ++inference.tokens_to_generate=120000 \ ++inference.temperature=0.6 \ ++inference.top_p=0.95 \ ++parse_reasoning=True ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (avg-of-8) 84.91% 3.54%
**majority\@8** **88.89%** 0.00%
pass\@8 96.46% 0.00%
#### 5.1.2 AIME 2025 * Dataset: AIME 2025 (30 problems) * Prompt: `generic/math` (boxed answer format) * Evaluation command: ```bash Command theme={null} ns prepare_data aime25 ns eval \ --cluster=local \ --server_type=openai \ --model=MiniMaxAI/MiniMax-M2.7 \ --server_address=http://localhost:30000/v1 \ --output_dir=./m2.7-eval/ \ --benchmarks=aime25:8 \ ++inference.tokens_to_generate=120000 \ ++inference.temperature=0.6 \ ++inference.top_p=0.95 \ ++parse_reasoning=True ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (avg-of-8) 92.50% ± 5.56% 2.92%
**majority\@8** **97.08%** 0.00%
pass\@8 100.00% 0.00%
#### 5.1.3 MMLU-Pro * Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice) * Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) * Evaluation command: ```bash Command theme={null} ns prepare_data mmlu-pro ns eval \ --cluster=local \ --server_type=openai \ --model=MiniMaxAI/MiniMax-M2.7 \ --server_address=http://localhost:30000/v1 \ --output_dir=./m2.7-eval/ \ --benchmarks=mmlu-pro \ ++prompt_config=eval/aai/mcq-10choices \ ++inference.tokens_to_generate=32768 \ ++inference.temperature=0.0 \ ++parse_reasoning=True ``` * Test Results:
Evaluation Mode Accuracy No Answer
pass\@1 (greedy) 69.41% 18.75%
> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly. #### 5.1.4 GSM8K Benchmark * Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API * Test Results: ```text Output theme={null} GSM8K Results (8-shot CoT) Model: MiniMaxAI/MiniMax-M2.7 Total: 1319 Correct: 1218 Accuracy: 92.34% ``` ### 5.2 Speed Benchmark #### 5.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 34.33 Total input tokens: 6101 Total generated tokens: 4220 Request throughput (req/s): 0.29 Input token throughput (tok/s): 177.71 Output token throughput (tok/s): 122.92 Total token throughput (tok/s): 300.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3431.21 Median E2E Latency (ms): 2742.57 ---------------Time to First Token---------------- Mean TTFT (ms): 50.28 Median TTFT (ms): 53.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.02 Median TPOT (ms): 8.01 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.03 Median ITL (ms): 8.02 ================================================== ``` #### 5.2.2 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model MiniMaxAI/MiniMax-M2.7 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 100.20 Total input tokens: 249831 Total generated tokens: 252662 Request throughput (req/s): 4.99 Input token throughput (tok/s): 2493.41 Output token throughput (tok/s): 2521.66 Total token throughput (tok/s): 5015.07 Concurrency: 90.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18072.69 Median E2E Latency (ms): 17761.84 ---------------Time to First Token---------------- Mean TTFT (ms): 247.94 Median TTFT (ms): 92.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.75 Median TPOT (ms): 36.67 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.34 Median ITL (ms): 30.55 ================================================== ``` # MiniMax-M3 Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3 Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from # the PR head. The serving runtime is in the base dependencies, so no extra is needed: git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/27944/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install --prerelease=allow sglang` will pull M3 support directly. ```bash Command theme={null} # Pull the M3 image the command panel selects for your platform, e.g.: docker pull lmsysorg/sglang:dev-cu13-minimax-m3 ``` The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces. These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path. Pick your hardware + recipe to generate the launch command. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **\~428B total parameters with \~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports \~9× prefill / \~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, \~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**. Key characteristics as served by SGLang: * **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here. * **Reasoning model**: emits its chain of thought wrapped in `...`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`. * **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported. * **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1). * **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized. **Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client. **Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA) ## 2. Configuration Tips ### 2.1 MSA sparse-attention fast path (recommended for Blackwell users) [MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops \~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs. **Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)): * **GPU**: NVIDIA SM100 family — sm\_100 (B200 / GB200) and sm\_103 (B300 / GB300). * **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import. * **Python**: ≥ 3.10; **OS**: Linux — works on both **x86\_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits. The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`. ```bash Command theme={null} # Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation git clone --recursive https://github.com/MiniMax-AI/MSA.git msa cd msa && pip install . # Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback): python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())" ``` The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache. **Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache. The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths. For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling. ### 2.2 Memory and workload tuning The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm\_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm\_100; GB300 is sm\_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**. * **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context. * **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated. * **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom. * **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy. * `--trust-remote-code` is required to load the MiniMax config / processor classes. ### 2.3 AMD Instinct (ROCm) MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way: * **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags. * **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card). Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path. The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path. ### 2.4 Serving on Hopper (H200) with the bf16 build The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal: * **MoE runner**: Triton, auto-selected for bf16 weights. * **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here. * **CUDA graph**: on, with full decode-graph capture. **High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **\~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving. Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4. ## 3. Advanced Usage ### 3.1 Reasoning Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ```text Output theme={null} =============== Reasoning =============== 15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36. =============== Answer ================== 15% of 240 is **36**. (10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.) ``` When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time: ```python Example theme={null} response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}], max_tokens=2048, stream=True, ) for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta if getattr(delta, "reasoning_content", None): print(delta.reasoning_content, end="", flush=True) # thinking stream if delta.content: print(delta.content, end="", flush=True) # answer stream print() ``` **Output Example:** ```text Output theme={null} [delta.reasoning_content — thinking stream] Let me solve this step by step. 15% of 240 = 0.15 × 240 = 36 Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓ [delta.content — answer stream] # Solving 15% of 240 ## Step 1: Convert the percentage to a decimal 15% = 15/100 = 0.15 ## Step 2: Multiply by 240 0.15 × 240 = 36 ## Answer **15% of 240 = 36** ``` ### 3.2 Tool Calling Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format: ```text Raw model output theme={null} ]<]minimax[>[ ]<]minimax[>[]<]minimax[>[Beijing]<]minimax[>[]<]minimax[>[ ]<]minimax[>[ ``` The parser converts that into the standard OpenAI `tool_calls` structure: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` ```text Output theme={null} Tool: get_weather Args: {"location": "Beijing"} ``` Beyond a single flat call, the parser also supports: * **Parallel calls** — multiple `` blocks inside the single `` wrapper, surfaced as multiple `message.tool_calls` entries. * **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object. * **Array arguments** — an `array`-typed parameter uses repeated `` children and is reconstructed into a JSON list. For example, a tool with object and array parameters round-trips cleanly: ```text Output theme={null} create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}} ``` To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both. ### 3.3 Multimodal (Vision) Input Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" }, }, {"type": "text", "text": "Describe this image in detail."}, ], } ], max_tokens=1024, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image captures a striking and unusual urban scene on what appears to be a busy New York City street. **Main Subject:** A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand. **Vehicles:** - The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform. - A second yellow taxi (a sedan) drives past on the left, captured with motion blur. **Setting:** Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan. ``` Notes: * If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side. * Multiple images per message are supported; add more `image_url` entries to the `content` list. * Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `` trace and/or tool calls. ### 3.4 Prefill-Decode (PD) Disaggregation [PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags. **Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only): * **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`. * **Single pipeline stage** — PP = 1 (the default). * **mooncake or NIXL** transfer backend over RDMA / InfiniBand. Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware: On Blackwell the MXFP8 recipe — fa4, page size 128, deep\_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3-MXFP8 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 4 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --disaggregation-ib-device mlx5_0 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3-MXFP8 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 4 \ --disaggregation-mode decode \ --disaggregation-transfer-backend nixl \ --disaggregation-ib-device mlx5_0 \ --host 0.0.0.0 --port 30001 ``` On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 8 \ --attention-backend triton \ --moe-runner-backend triton \ --page-size 128 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-M3 \ --trust-remote-code \ --reasoning-parser auto \ --tool-call-parser auto \ --tp 8 \ --attention-backend triton \ --moe-runner-backend triton \ --page-size 128 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30001 ``` Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint: ```bash PD router theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30001 \ --policy round_robin \ --host 0.0.0.0 --port 8000 ``` Clients hit the router exactly like a single server — it splits each request across the two stages transparently: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="MiniMaxAI/MiniMax-M3-MXFP8", messages=[{"role": "user", "content": "What is 2 + 2?"}], max_tokens=64, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} 2 + 2 = 4 ``` **Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy. * **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate). * **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving. # Devstral 2 (Mistral) Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Devstral-2 ## 1. Model Introduction **Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**. The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads. **Key Features:** * **Agentic coding**: Optimized for tool-driven coding and software engineering agents * **Improved performance**: A step up compared to earlier Devstral models * **Better generalization**: More robust across diverse prompts and coding environments * **Long context**: Up to a **256K** context window **Use Cases:** AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration. For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral. **Models:** * **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2) * **FP8 Instruct**: * **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)** * **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)** *** ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`: ```shell Command theme={null} python -c "import transformers; print(transformers.__version__)" ``` If your version is lower, upgrade: ```shell Command theme={null} pip install -U --pre "transformers>=5.0.0rc0" ``` *** ## 3. Model Deployment ### 3.1 Basic configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B). The TP size is set to the minimum required for the selected model size. ### 3.2 Configuration tips * **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable. * **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers. *** ## 4. Model Invocation ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang exposes an OpenAI-compatible endpoint. Example: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="mistralai/Devstral-Small-2-24B-Instruct-2512", messages=[ {"role": "system", "content": "You are a helpful coding assistant."}, {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."}, ], temperature=0.2, max_tokens=512, ) print(resp.choices[0].message.content) ``` **Output Example:** ````text Output theme={null} Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors. ```python import time import requests from requests.exceptions import RequestException def retry_with_exponential_backoff( url, max_retries=3, initial_delay=1, backoff_factor=2, method="GET", **kwargs ): """ Retry a request with exponential backoff. Parameters: - url: The URL to request. - max_retries: Maximum number of retry attempts (default: 3). - initial_delay: Initial delay in seconds (default: 1). - backoff_factor: Multiplier for the delay between retries (default: 2). - method: HTTP method to use (default: "GET"). - **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.). Returns: - Response object if the request succeeds. - Raises an exception if all retries fail. """ retry_count = 0 delay = initial_delay while retry_count < max_retries: try: response = requests.request(method, url, **kwargs) # Check if the response status code indicates success if response.status_code < 400: return response else: raise RequestException(f"HTTP {response.status_code}: {response.text}") except RequestException as e: if retry_count == max_retries - 1: raise Exception(f"All retries failed. Last error: {e}") print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...") time.sleep(delay) ... ```` ### 4.2 Tool calling (optional) Devstral 2 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --tp 2 \ --tool-call-parser mistral ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="mistralai/Devstral-2-123B-Instruct-2512", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing"} ``` ## AMD GPU Support ## 1. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 1.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 1.2 Advanced Usage ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path mistralai/Devstral-2-123B-Instruct-2512 \ --tp 8 \ --trust-remote-code \ --port 8888 ``` ## 2.Benchmark ### 5.1 Benchmark Commands **Scenario 1: Chat (1K/1K) - Most Important** * **Model Deployment** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path mistralai/Devstral-2-123B-Instruct-2512 \ --tp 8 \ --trust-remote-code \ --port 8888 ``` * Low Concurrency (Latency-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 94.30 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4206 Request throughput (req/s): 0.11 Input token throughput (tok/s): 64.70 Output token throughput (tok/s): 44.75 Peak output token throughput (tok/s): 82.00 Peak concurrent requests: 2 Total token throughput (tok/s): 109.44 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9427.59 Median E2E Latency (ms): 5637.23 ---------------Time to First Token---------------- Mean TTFT (ms): 4253.85 Median TTFT (ms): 116.95 P99 TTFT (ms): 37764.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 12.28 Median TPOT (ms): 12.29 P99 TPOT (ms): 12.30 ---------------Inter-Token Latency---------------- Mean ITL (ms): 12.29 Median ITL (ms): 12.29 P95 ITL (ms): 12.38 P99 ITL (ms): 12.42 Max ITL (ms): 12.90 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 52.11 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40761 Request throughput (req/s): 1.54 Input token throughput (tok/s): 761.31 Output token throughput (tok/s): 783.13 Peak output token throughput (tok/s): 1120.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1544.44 Concurrency: 13.60 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8856.19 Median E2E Latency (ms): 9314.71 ---------------Time to First Token---------------- Mean TTFT (ms): 398.80 Median TTFT (ms): 127.81 P99 TTFT (ms): 1500.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.32 Median TPOT (ms): 16.90 P99 TPOT (ms): 32.78 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.61 Median ITL (ms): 14.26 P95 ITL (ms): 15.07 P99 ITL (ms): 114.46 Max ITL (ms): 1224.45 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model mistralai/Devstral-2-123B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf \ --port 8888 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 116.08 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252523 Request throughput (req/s): 4.31 Input token throughput (tok/s): 2152.21 Output token throughput (tok/s): 2176.60 Peak output token throughput (tok/s): 3600.00 Peak concurrent requests: 107 Total token throughput (tok/s): 4328.81 Concurrency: 92.42 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21456.71 Median E2E Latency (ms): 20126.82 ---------------Time to First Token---------------- Mean TTFT (ms): 291.60 Median TTFT (ms): 199.24 P99 TTFT (ms): 866.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 42.42 Median TPOT (ms): 45.18 P99 TPOT (ms): 53.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 41.97 Median ITL (ms): 27.59 P95 ITL (ms): 130.43 P99 ITL (ms): 137.87 Max ITL (ms): 616.73 ================================================== ``` #### 5.2 Understanding the Results **Key Metrics:** * **Request Throughput (req/s)**: Number of requests processed per second * **Output Token Throughput (tok/s)**: Total tokens generated per second * **Mean TTFT (ms)**: Time to First Token - measures responsiveness * **Mean TPOT (ms)**: Time Per Output Token - measures generation speed * **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency **Why These Configurations Matter:** * **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments. * **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations. * **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks. * **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput. **Interpreting Results:** * Compare your results against baseline numbers for your hardware * Higher throughput at same latency = better performance * Lower TTFT = more responsive user experience * Lower TPOT = faster generation speed ### 5.3 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.3.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py \ --num-shots 8 \ --num-questions 1316 \ --parallel 1316 \ --port 8888 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.922 Invalid: 0.000 Latency: 35.800 s Output throughput: 4507.697 token/s ``` # Ministral-3 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Ministral-3 ## 1. Model Introduction The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities. The Ministral 3 14B Instruct model offers the following capabilities: Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text. Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic. System Prompt: Maintains strong adherence and support for system prompts. Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting. Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere. Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes. Large Context Window: Supports a 256k context window. For further details, please refer to the [official documentation](https://github.com/mistralai) ## 2. SGLang Installation Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ### 3.2 Configuration Tips **Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable. **Pre-installation steps**: Adding the following steps after launching the docker ```shell Command theme={null} pip install mistral-common --upgrade pip install transformers==5.0.0.rc0 ``` ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Ministral \ lmsysorg/sglang:v0.5.9-rocm720-mi30x \ /bin/bash ``` #### 4.2.2 Launch the server ```shell Command theme={null} sglang serve \ --model-path mistralai/Ministral-3-14B-Instruct-2512 \ --tp 1 \ --trust-remote-code ``` ## 5. Benchmark This section uses **industry-standard configurations** for comparable benchmark results. ### 5.1 Speed Benchmark **Test Environment:** * Hardware: MI300X GPU (8x) * Model: mistralai/Ministral-3-14B-Instruct-2512 * Tensor Parallelism: 1 * SGLang Version: 0.5.7 * Model Deployment Command: ```bash Command theme={null} sglang serve \ --model-path mistralai/Ministral-3-14B-Instruct-2512 \ --tp 1 \ --trust-remote-code ``` ##### Low Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 65.08 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.15 Input token throughput (tok/s): 93.75 Output token throughput (tok/s): 64.84 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 158.59 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6505.51 Median E2E Latency (ms): 3037.37 ---------------Time to First Token---------------- Mean TTFT (ms): 3709.33 Median TTFT (ms): 53.72 P99 TTFT (ms): 33320.77 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.63 Median TPOT (ms): 6.64 P99 TPOT (ms): 6.66 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.64 Median ITL (ms): 6.65 P95 ITL (ms): 6.75 P99 ITL (ms): 6.82 Max ITL (ms): 8.45 ================================================== ``` ##### Medium Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 31.20 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 40783 Request throughput (req/s): 2.56 Input token throughput (tok/s): 1271.38 Output token throughput (tok/s): 1307.82 Peak output token throughput (tok/s): 1760.00 Peak concurrent requests: 22 Total token throughput (tok/s): 2579.20 Concurrency: 13.72 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5351.07 Median E2E Latency (ms): 5626.45 ---------------Time to First Token---------------- Mean TTFT (ms): 280.87 Median TTFT (ms): 68.16 P99 TTFT (ms): 1194.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.47 Median TPOT (ms): 10.10 P99 TPOT (ms): 20.00 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.96 Median ITL (ms): 9.10 P95 ITL (ms): 9.87 P99 ITL (ms): 51.39 Max ITL (ms): 888.63 ================================================== ``` ##### High Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model mistralai/Ministral-3-14B-Instruct-2512 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 88.75 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 252547 Request throughput (req/s): 5.63 Input token throughput (tok/s): 2815.01 Output token throughput (tok/s): 2846.91 Peak output token throughput (tok/s): 4271.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5661.93 Concurrency: 93.04 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16514.45 Median E2E Latency (ms): 15834.45 ---------------Time to First Token---------------- Mean TTFT (ms): 148.57 Median TTFT (ms): 99.15 P99 TTFT (ms): 455.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.93 Median TPOT (ms): 34.73 P99 TPOT (ms): 38.05 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.45 Median ITL (ms): 27.30 P95 ITL (ms): 71.73 P99 ITL (ms): 73.45 Max ITL (ms): 328.10 ================================================== ``` ### 5.2 Accuracy Benchmark Document model accuracy on standard benchmarks: #### 5.2.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py \ --num-shots 8 \ --num-questions 1316 \ --parallel 1316 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.959 Invalid: 0.000 Latency: 29.185 s Output throughput: 4854.672 token/s ``` # Mistral Medium 3.5 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Medium-3.5 ## 1. Model Introduction **Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios. **Key Features:** * **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head\_dim=128) * **256K context window** — YARN RoPE scaling on top of the original 4K base * **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`) * **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios * **Function Calling**: Native tool calling and JSON output * **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in * **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali * **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue) **Architecture:** * Mistral 3 backbone with YARN RoPE for 256K context * Dense (no MoE), 128B parameters * Standard GQA attention (not MLA) * Pixtral-style vision encoder (48 layers, patch\_size=14, spatial\_merge=2, image\_size=1540) trained from scratch * Multimodal input: text + image **Models:** * **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8) The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present. *** ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). **Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300). *** ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5. ### 3.2 Configuration Tips * **Tensor Parallelism**: Mistral Medium 3.5 FP8 (\~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300). * **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call. * **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output. * **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable. * **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support. * **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content. * **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6). ### 3.3 Speculative Decoding (EAGLE) Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (\~4 GB), and is meant for low-concurrency latency-bound serving. ```bash Command theme={null} python -m sglang.launch_server \ --model-path mistralai/Mistral-Medium-3.5-128B \ --tp 4 \ --dtype bfloat16 \ --tool-call-parser mistral \ --reasoning-parser mistral \ --speculative-algorithm EAGLE \ --speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --port 30000 ``` * **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16). * The draft uses the same vocab and lm\_head as the target. Memory overhead on top of the base model is \~4 GB per TP shard. * `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text. * EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads. *** ## 4. Model Invocation ### 4.1 Thinking Mode Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"}, ], temperature=0.7, extra_body={"reasoning_effort": "high"}, ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Answer:", response.choices[0].message.content) ``` **Output:** ```text Output theme={null} Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and division before addition, evaluated left to right. 17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391. 144 / 12 = 12. Finally, 391 + 12 = 403. Answer: **17 × 23 + 144 / 12 = 403** Step by step: 1. 17 × 23 = 391 2. 144 / 12 = 12 3. 391 + 12 = 403 ``` ### 4.2 Instruct Mode (Reasoning Off) To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "What is the capital of France?"}, ], temperature=0.1, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The capital of France is **Paris**. It is one of the most famous and visited cities in the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral. ``` ### 4.3 Streaming with Reasoning ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ {"role": "user", "content": "Explain the difference between async and threading in Python."}, ], temperature=0.7, extra_body={"reasoning_effort": "high"}, stream=True, ) print("=== Reasoning ===") for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) elif delta.content: print("\n=== Response ===") print(delta.content, end="", flush=True) print() ``` ### 4.4 Tool Calling Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls for tc in tool_calls: print(f"Tool: {tc.function.name}") print(f"Args: {tc.function.arguments}") ``` **Output:** ```text Output theme={null} Tool: get_weather Args: {"location": "Paris"} ``` ### 4.5 Vision (Image Input) Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Medium-3.5-128B", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe what you see in this image."}, { "type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"}, }, ], } ], temperature=0.7, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The image features a stylized representation of the acronym "SGL." The letters are large, bold, and orange with a brown outline, giving them a three-dimensional effect. To the left of the letters, there is a graphic that resembles a neuron or a node with connections, also in a similar orange and brown color scheme. The node has a code symbol () inside a square, suggesting a connection to programming or technology. ``` ### 4.6 Loading the Reference System Prompt Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly: ```python Example theme={null} from datetime import datetime, timedelta from huggingface_hub import hf_hub_download from openai import OpenAI MODEL = "mistralai/Mistral-Medium-3.5-128B" def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str: path = hf_hub_download(repo_id=repo_id, filename=filename) today = datetime.today().strftime("%Y-%m-%d") yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d") name = repo_id.split("/")[-1] with open(path) as f: return f.read().format(name=name, today=today, yesterday=yesterday) client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model=MODEL, messages=[ {"role": "system", "content": load_system_prompt(MODEL)}, {"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."}, ], temperature=0.1, extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` *** ## 5. Benchmarks Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint. ### 5.1 Accuracy Benchmarks #### GSM8K ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` **Results:** ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 13.594 s Output throughput: 1560.660 token/s ``` #### MMMU ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --port 30000 ``` **Results:** ```text Output theme={null} Overall accuracy: 0.586 ``` ### 5.2 Speed Benchmarks #### Latency (Low Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 10 Benchmark duration (s): 38.86 Total input tokens: 6101 Total generated tokens: 2684 Output token throughput (tok/s): 69.07 Mean E2E Latency (ms): 3883.80 Median TTFT (ms): 95.90 Median TPOT (ms): 14.19 ================================================== ``` #### Throughput (High Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 1000 \ --max-concurrency 100 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 1000 Benchmark duration (s): 117.28 Total input tokens: 512842 Total generated tokens: 262023 Output token throughput (tok/s): 2234.18 Total token throughput (tok/s): 6607.01 Mean E2E Latency (ms): 11303.79 Median TTFT (ms): 152.95 Median TPOT (ms): 42.53 ================================================== ``` ### 5.3 EAGLE Speculative Decoding (Latency) Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`). ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Successful requests: 10 Benchmark duration (s): 27.64 Total input tokens: 6101 Total generated tokens: 2684 Output token throughput (tok/s): 97.10 Mean E2E Latency (ms): 2762.99 Median TTFT (ms): 90.69 Median TPOT (ms): 9.73 Accept length: 1.72 ================================================== ``` EAGLE delivers **\~1.41× output throughput and \~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens. # Mistral Small 4 Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Small-4 ## 1. Model Introduction **Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model. With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3. **Key Features:** * **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request * **Vision**: Accepts both text and image inputs, providing insights based on visual content * **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities * **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more * **Context Window**: 256K context window * **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters) * **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes * Reasoning effort supported are only **"none" and "high"** **Architecture:** * Same general architecture as Mistral 3 * MoE: 128 experts, 4 active per token * 119B total parameters, 6.5B activated per token * Multimodal input: text + image **Models:** * **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8) * **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)** * **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603 * **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference *** ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/install). *** ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4. ### 3.2 Configuration Tips * **Tensor Parallelism**: Mistral Small 4 FP8 (\~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (\~60 GB, Blackwell only) runs with tp=1. * **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call. * **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable. * **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support. * **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content. * **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency. *** ## 4. Model Invocation ### 4.1 Thinking Mode Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"}, ], extra_body={"reasoning_effort": "high"}, ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Answer:", response.choices[0].message.content) ``` **Output:** ```text Output theme={null} Reasoning: First, I'll break down the problem into two parts: the multiplication and the division. According to the order of operations (PEMDAS/BODMAS), multiplication and division are performed from left to right before addition. 17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391 144 / 12 = 12 Finally, add the results: 391 + 12 = 403 Answer: The solution to the problem is as follows: 1. First, perform the multiplication: 17 × 23. - 17 × 20 = 340 - 17 × 3 = 51 - 340 + 51 = 391 2. Then, perform the division: 144 / 12 = 12. 3. Finally, add the results: - 391 + 12 = 403 **Answer:** \boxed{403} ``` ### 4.2 Instruct Mode (Reasoning Off) To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Write a Python function to reverse a string."}, ], extra_body={"reasoning_effort": "none"}, ) print(response.choices[0].message.content) ``` **Output:** ````text Output theme={null} # Python Function to Reverse a String Here are several ways to write a Python function to reverse a string: ## Method 1: Using String Slicing (Most Pythonic) ```python def reverse_string(s): """Reverse a string using slicing.""" return s[::-1] ``` ## Method 2: Using a Loop ```python Example def reverse_string(s): """Reverse a string using a loop.""" reversed_str = "" for char in s: reversed_str = char + reversed_str return reversed_str ``` ## Method 3: Using reversed() function ```python Example def reverse_string(s): """Reverse a string using reversed() function.""" return ''.join(reversed(s)) ``` The first method using string slicing (`s[::-1]`) is generally the most efficient and recommended approach in Python. Example usage: ```python Example original = "Hello, World!" reversed_str = reverse_string(original) print(reversed_str) # Output: "!dlroW ,olleH" ``` ```` ### 4.3 Streaming with Reasoning ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ {"role": "user", "content": "Explain the difference between async and threading in Python."}, ], extra_body={"reasoning_effort": "high"}, stream=True, ) print("=== Reasoning ===") for chunk in stream: delta = chunk.choices[0].delta if hasattr(delta, "reasoning_content") and delta.reasoning_content: print(delta.reasoning_content, end="", flush=True) elif delta.content: print("\n=== Response ===") print(delta.content, end="", flush=True) print() ``` **Output:** ```text Output theme={null} === Reasoning === Okay, the user is asking about the difference between async and threading in Python. I need to break this down clearly, covering the key aspects of both, like their purposes, performance characteristics, and use cases... === Response === In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency models, each suited for specific use cases. Here's a breakdown of their key differences: ### 1. Model of Concurrency - **Threading**: Based on preemptive multitasking using OS threads. - **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield... ``` ### 4.4 Tool Calling Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[{"role": "user", "content": "What's the weather in Paris?"}], tools=tools, tool_choice="auto", ) tool_calls = response.choices[0].message.tool_calls for tc in tool_calls: print(f"Tool: {tc.function.name}") print(f"Args: {tc.function.arguments}") ``` **Output:** ```text Output theme={null} Tool: get_weather Args: {"location": "Paris"} ``` ### 4.5 Vision (Image Input) Mistral Small 4 accepts image inputs alongside text: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="mistralai/Mistral-Small-4-119B-2603", messages=[ { "role": "user", "content": [ {"type": "text", "text": "Describe what you see in this image."}, { "type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"}, }, ], } ], ) print(response.choices[0].message.content) ``` **Output:** ```text Output theme={null} The image is a copyright symbol, represented by a stylized version of the lowercase letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and the circle is orange. The design is simple yet striking, using oval and elliptical shapes to create a distinct symbol which signifies copyright protection. ``` *** ## 5. Benchmarks ### 5.1 Accuracy Benchmarks #### GSM8K ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` #### MMLU ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` ### 5.2 Speed Benchmarks #### Latency (Low Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompts 10 \ --max-concurrency 1 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` #### Throughput (High Concurrency) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompts 1000 \ --max-concurrency 100 \ --random-input-len 1024 \ --random-output-len 512 \ --port 30000 ``` **Results:** ```text Output theme={null} TODO ``` # Kimi-K2 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2 ## 1. Model Introduction [Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters. **Model Variants:** * **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM. * **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use. * **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified). For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory. * **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism). * **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput. * **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+). * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/get-started/quickstart). ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Enable reasoning parser for Kimi-K2-Thinking: ```shell Command theme={null} python -m sglang.launch_server \ --model moonshotai/Kimi-K2-Thinking \ --reasoning-parser kimi_k2 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.6, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step. Step 1: Understand what "percent" means. - "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15. Step 2: Convert the percentage to a decimal. - 15% = 15 / 100 = 0.15 Step 3: Multiply the decimal by the number. - 0.15 * 240 Step 4: Perform the multiplication. - 0.15 * 240 = (15/100) * 240 - = 15 * 240 / 100 - = 3600 / 100 - = 36 Alternatively, I can calculate it directly: - 0.15 * 240 - 15 * 240 = 3600 - 3600 / 100 = 36 Or, break it down: - 10% of 240 = 24 - 5% of 240 = half of 10% = 12 - 15% of 240 = 10% + 5% = 24 + 12 = 36 I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying. Let me structure the answer: 1. Convert the percentage to a decimal. 2. Multiply the decimal by the number. 3. Show the calculation. 4. State the final answer. This is simple and easy to follow. =============== Content ================= Here is the step-by-step solution: **Step 1: Convert the percentage to a decimal** 15% means 15 per 100, which is 15 ÷ 100 = **0.15** **Step 2: Multiply the decimal by the number** 0.15 × 240 **Step 3: Calculate the result** 0.15 × 240 = **36** **Answer:** 15% of 240 is **36**. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.2 Tool Calling Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment: **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model moonshotai/Kimi-K2-Instruct \ --tool-call-parser kimi_k2 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location":"Beijing"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Kimi-K2-Instruct * sglang version: 0.5.6.post1 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --dp 4 \ --enable-dp-attention \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model moonshotai/Kimi-K2-Instruct\ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 44.93 Total input tokens: 1951 Total input text tokens: 1951 Total input vision tokens: 0 Total generated tokens: 2755 Total generated tokens (retokenized): 2748 Request throughput (req/s): 0.22 Input token throughput (tok/s): 43.42 Output token throughput (tok/s): 61.32 Peak output token throughput (tok/s): 64.00 Peak concurrent requests: 3 Total token throughput (tok/s): 104.74 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4489.56 Median E2E Latency (ms): 4994.53 ---------------Time to First Token---------------- Mean TTFT (ms): 141.22 Median TTFT (ms): 158.28 P99 TTFT (ms): 166.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.40 Median TPOT (ms): 15.63 P99 TPOT (ms): 39.88 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.78 Median ITL (ms): 15.76 P95 ITL (ms): 16.36 P99 ITL (ms): 16.59 Max ITL (ms): 19.94 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --dp 4 \ --ep 4 \ --enable-dp-attention \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 8000 \ --model moonshotai/Kimi-K2-Instruct\ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results**: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 174.11 Total input tokens: 296642 Total input text tokens: 296642 Total input vision tokens: 0 Total generated tokens: 193831 Total generated tokens (retokenized): 168687 Request throughput (req/s): 5.74 Input token throughput (tok/s): 1703.73 Output token throughput (tok/s): 1113.25 Peak output token throughput (tok/s): 2383.00 Peak concurrent requests: 112 Total token throughput (tok/s): 2816.97 Concurrency: 89.60 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15601.09 Median E2E Latency (ms): 10780.52 ---------------Time to First Token---------------- Mean TTFT (ms): 457.42 Median TTFT (ms): 221.62 P99 TTFT (ms): 2475.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 97.23 Median TPOT (ms): 85.61 P99 TPOT (ms): 435.95 ---------------Inter-Token Latency---------------- Mean ITL (ms): 78.61 Median ITL (ms): 43.66 P95 ITL (ms): 169.53 P99 ITL (ms): 260.91 Max ITL (ms): 1703.21 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Server Command ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-K2-Instruct \ --tp 8 \ --dp 4 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 ``` * Benchmark Command ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000 ``` * **Result**: ```text Output theme={null} Accuracy: 0.960 Invalid: 0.000 Latency: 15.956 s Output throughput: 1231.699 token/s ``` # Kimi-K2.5 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.5 ## 1. Model Introduction [Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) is an open-source, native multimodal agentic model by Moonshot AI, built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes. **Key Features:** * **Native Multimodality**: Pre-trained on vision-language tokens, K2.5 excels in visual knowledge, cross-modal reasoning, and agentic tool use grounded in visual inputs. * **Coding with Vision**: K2.5 generates code from visual specifications (UI designs, video workflows) and autonomously orchestrates tools for visual data processing. * **Agent Swarm**: K2.5 transitions from single-agent scaling to a self-directed, coordinated swarm-like execution scheme. It decomposes complex tasks into parallel sub-tasks executed by dynamically instantiated, domain-specific agents. * **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency. **Available Models**: * INT4 (Initial Released): [moonshotai/Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) * NVFP4 (4-bit quantized): [nvidia/Kimi-K2.5-NVFP4](https://huggingface.co/nvidia/Kimi-K2.5-NVFP4) For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.5) and [deployment guidance](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/docs/deploy_guidance.md). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with >=140GB each. Supported platforms: H200 (8x, TP=8), B300 (8x, TP=8), GB300 (4x, TP=4), MI300X/MI325X (4x, TP=4), MI350X/MI355X (4x, TP=4). Use `--context-length 128000` to conserve memory. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be \<= 4 (not 8). Kimi-K2.5 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. The ROCm 7.2 images (`rocm720`) have an AITER compatibility issue. * **DP Attention**: Enable with `--dp --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.5 supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **receipt from Auntie Anne's** (a pretzel franchise restaurant). ## Key Details: **Item Purchased:** - **CINNAMON SUGAR** - 1 unit x 17,000 = **17,000** **Payment Summary:** - **SUB TOTAL:** 17,000 - **GRAND TOTAL:** 17,000 - **CASH IDR:** 20,000 (Indonesian Rupiah) - **CHANGE DUE:** 3,000 ## Context: The receipt indicates a transaction in **Indonesian Rupiah (IDR)**. A customer purchased one Cinnamon Sugar pretzel for 17,000 IDR, paid with a 20,000 IDR note, and received 3,000 IDR in change. The top of the receipt shows the Auntie Anne's logo (a heart-shaped pretzel with a halo), and some text appears blurred for privacy, likely obscuring the store location, date, and transaction number. The receipt is printed on white thermal paper. ``` #### 4.2.2 Reasoning Output Kimi-K2.5 supports both thinking mode (default) and instant mode. **Thinking Mode (default)** -- reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Instant Mode (thinking off)** -- disable thinking for faster responses: ```python Example theme={null} response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print("====== Response (Instant Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. At first glance, someone might think 9.11 is bigger because 11 > 9, but that's incorrect because we're dealing with decimal numbers, not whole numbers. Let me compare them properly: - 9.9 = 9.90 - 9.11 When comparing decimals, we look at each place value from left to right: - Units place: 9 = 9 (tie) - Tenths place: 9 vs 1 Since 9 > 1, we have 9.9 > 9.11. Alternatively, we can think of it as: - 9.9 = 9 + 9/10 = 9 + 0.9 = 9.90 - 9.11 = 9 + 11/100 = 9 + 0.11 Since 0.90 > 0.11, then 9.9 > 9.11. So the answer is clearly 9.9 is bigger. The "think carefully" hint suggests the user is trying to catch the common error where people compare 11 and 9 as whole numbers rather than understanding decimal place value (tenths vs hundredths). I should explain this clearly to avoid confusion. ====== Response (Thinking Mode) ====== **9.9 is bigger.** Here's why this can be tricky: Many people instinctively compare 11 and 9 and think "11 is bigger than 9," but that's comparing the wrong place values. When comparing decimals, align them by place value: - 9.9 = 9.**90** - 9.11 = 9.**11** After the decimal point: - The first digit (tenths place): **9** vs **1** - Since 9 > 1, we stop there. **9.9 is larger.** Think of it as money: - $9.90 (nine dollars and ninety cents) - $9.11 (nine dollars and eleven cents) $9.90 is clearly more than $9.11. ====== Response (Instant Mode) ====== Let me think through this carefully. **9.9 is bigger than 9.11** Here's why: When comparing decimals, we need to align them by their decimal places: - 9.9 = 9.90 - 9.11 = 9.11 Now comparing: - The whole number parts are equal (9 = 9) - Comparing tenths: **9 > 1** So 9.90 > 9.11 A common mistake is thinking 11 hundredths is larger than 9 tenths, but 9 tenths = 90 hundredths, which is clearly larger than 11 hundredths. ``` #### 4.2.3 Tool Calling Kimi-K2.5 supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is **22°C and sunny**. ☀️ It's a nice day there with comfortable temperatures and clear skies! ``` #### 4.2.4 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.5", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user is asking me to identify a product from a receipt and search for similar items. Looking at the receipt, I can see: 1. The store is "Auntie Anne's" - which is a popular pretzel chain 2. The product purchased is "CINNAMON SUGAR" 3. Price is 17,000 (likely Indonesian Rupiah based on "CASH IDR") 4. Quantity is 1 So the product is a Cinnamon Sugar pretzel from Auntie Anne's. Now I need to search for this product or similar items using the search_product function. === Content === I can see from the receipt that the product is a **Cinnamon Sugar** item from **Auntie Anne's** (the famous pretzel chain). This appears to be a Cinnamon Sugar Pretzel purchased for 17,000 IDR (Indonesian Rupiah). Let me search for this product and similar items: === Tool Calls === Function: search_product Arguments: {"query": "Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.5 Speculative Decoding **Nvidia** Deploy Kimi-K2.5 with the following command (H200/B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` Deploy Kimi-K2.5-NVFP4 with the following command (B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path nvidia/Kimi-K2.5-NVFP4 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --kv-cache-dtype fp8_e4m3 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU benchmark, which tests multimodal understanding and reasoning across various subjects: * **Benchmark Command:** ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --response-answer-regex "(?i)(?:answer|ans)[:\s]*(?:\*\*)?[\(\[]?([A-Za-z])[\)\]]?(?:\*\*)?" \ --port 30000 \ --concurrency 64 ``` * **Result:** ```text Output theme={null} Benchmark time: 2785.4322692090645 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.667, 'num': 30}, 'Agriculture': {'acc': 0.567, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.733, 'num': 30}, 'Art': {'acc': 0.833, 'num': 30}, 'Art_Theory': {'acc': 0.8, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.833, 'num': 30}, 'Biology': {'acc': 0.6, 'num': 30}, 'Chemistry': {'acc': 0.633, 'num': 30}, 'Clinical_Medicine': {'acc': 0.733, 'num': 30}, 'Computer_Science': {'acc': 0.667, 'num': 30}, 'Design': {'acc': 0.7, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.5, 'num': 30}, 'Economics': {'acc': 0.867, 'num': 30}, 'Electronics': {'acc': 0.3, 'num': 30}, 'Energy_and_Power': {'acc': 0.767, 'num': 30}, 'Finance': {'acc': 0.833, 'num': 30}, 'Geography': {'acc': 0.667, 'num': 30}, 'History': {'acc': 0.767, 'num': 30}, 'Literature': {'acc': 0.767, 'num': 30}, 'Manage': {'acc': 0.733, 'num': 30}, 'Marketing': {'acc': 0.833, 'num': 30}, 'Materials': {'acc': 0.567, 'num': 30}, 'Math': {'acc': 0.633, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.567, 'num': 30}, 'Music': {'acc': 0.5, 'num': 30}, 'Overall': {'acc': 0.698, 'num': 900}, 'Overall-Art and Design': {'acc': 0.708, 'num': 120}, 'Overall-Business': {'acc': 0.787, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.74, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.75, 'num': 120}, 'Overall-Science': {'acc': 0.66, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.595, 'num': 210}, 'Pharmacy': {'acc': 0.767, 'num': 30}, 'Physics': {'acc': 0.767, 'num': 30}, 'Psychology': {'acc': 0.667, 'num': 30}, 'Public_Health': {'acc': 0.867, 'num': 30}, 'Sociology': {'acc': 0.8, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.698 ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (8x) * Model: Kimi-K2.5 * Tensor Parallelism: 8 * SGLang Version: 0.5.6.post2 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. #### 5.2.1 Latency Benchmark * **Model Deployment:** ```bash Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.77 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4221 Request throughput (req/s): 0.25 Input token throughput (tok/s): 153.40 Output token throughput (tok/s): 106.10 Peak output token throughput (tok/s): 156.00 Peak concurrent requests: 2 Total token throughput (tok/s): 259.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3972.87 Median E2E Latency (ms): 4044.55 P90 E2E Latency (ms): 7046.30 P99 E2E Latency (ms): 7441.13 ---------------Time to First Token---------------- Mean TTFT (ms): 176.89 Median TTFT (ms): 154.24 P99 TTFT (ms): 285.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.22 Median TPOT (ms): 9.32 P99 TPOT (ms): 12.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.02 Median ITL (ms): 8.80 P95 ITL (ms): 13.23 P99 ITL (ms): 14.17 Max ITL (ms): 29.38 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 158.05 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 0.51 Input token throughput (tok/s): 250.99 Output token throughput (tok/s): 258.18 Peak output token throughput (tok/s): 1103.00 Peak concurrent requests: 19 Total token throughput (tok/s): 509.17 Concurrency: 14.09 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27837.05 Median E2E Latency (ms): 23508.00 P90 E2E Latency (ms): 57126.31 P99 E2E Latency (ms): 66044.35 ---------------Time to First Token---------------- Mean TTFT (ms): 374.30 Median TTFT (ms): 375.51 P99 TTFT (ms): 695.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.25 Median TPOT (ms): 57.93 P99 TPOT (ms): 85.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.95 Median ITL (ms): 53.97 P95 ITL (ms): 84.74 P99 ITL (ms): 244.84 Max ITL (ms): 655.61 ================================================== ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 996.64 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252588 Request throughput (req/s): 0.50 Input token throughput (tok/s): 250.67 Output token throughput (tok/s): 253.51 Peak output token throughput (tok/s): 1199.00 Peak concurrent requests: 104 Total token throughput (tok/s): 504.18 Concurrency: 92.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 184773.75 Median E2E Latency (ms): 174183.65 P90 E2E Latency (ms): 343625.28 P99 E2E Latency (ms): 404284.53 ---------------Time to First Token---------------- Mean TTFT (ms): 1289.59 Median TTFT (ms): 1313.35 P99 TTFT (ms): 2346.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 364.70 Median TPOT (ms): 403.32 P99 TPOT (ms): 452.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 363.82 Median ITL (ms): 316.21 P95 ITL (ms): 745.91 P99 ITL (ms): 1345.88 Max ITL (ms): 3118.59 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 680.26 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.01 Input token throughput (tok/s): 8.97 Output token throughput (tok/s): 65.36 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.33 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 68019.29 Median E2E Latency (ms): 70568.85 P90 E2E Latency (ms): 113237.40 P99 E2E Latency (ms): 121682.34 ---------------Time to First Token---------------- Mean TTFT (ms): 206.17 Median TTFT (ms): 177.28 P99 TTFT (ms): 445.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.36 Median TPOT (ms): 15.89 P99 TPOT (ms): 16.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.26 Median ITL (ms): 15.85 P95 ITL (ms): 17.50 P99 ITL (ms): 23.21 Max ITL (ms): 45.22 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 2475.98 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 318166 Request throughput (req/s): 0.03 Input token throughput (tok/s): 16.02 Output token throughput (tok/s): 128.56 Peak output token throughput (tok/s): 847.00 Peak concurrent requests: 18 Total token throughput (tok/s): 144.58 Concurrency: 14.62 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 452592.46 Median E2E Latency (ms): 486002.05 P90 E2E Latency (ms): 833197.57 P99 E2E Latency (ms): 957399.48 ---------------Time to First Token---------------- Mean TTFT (ms): 359.38 Median TTFT (ms): 350.78 P99 TTFT (ms): 500.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 111.18 Median TPOT (ms): 122.76 P99 TPOT (ms): 145.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 113.69 Median ITL (ms): 122.81 P95 ITL (ms): 147.87 P99 ITL (ms): 151.03 Max ITL (ms): 272.05 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} Waiting for completion... ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 120.73 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.08 Input token throughput (tok/s): 347.41 Output token throughput (tok/s): 34.96 Peak output token throughput (tok/s): 73.00 Peak concurrent requests: 2 Total token throughput (tok/s): 382.36 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12068.56 Median E2E Latency (ms): 10211.36 P90 E2E Latency (ms): 23203.32 P99 E2E Latency (ms): 30677.66 ---------------Time to First Token---------------- Mean TTFT (ms): 1625.64 Median TTFT (ms): 1526.63 P99 TTFT (ms): 3743.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.95 Median TPOT (ms): 23.95 P99 TPOT (ms): 35.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.80 Median ITL (ms): 21.73 P95 ITL (ms): 59.56 P99 ITL (ms): 61.10 Max ITL (ms): 62.70 ================================================== ``` * Medium Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 389.96 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41670 Request throughput (req/s): 0.21 Input token throughput (tok/s): 769.36 Output token throughput (tok/s): 106.86 Peak output token throughput (tok/s): 304.00 Peak concurrent requests: 19 Total token throughput (tok/s): 876.22 Concurrency: 14.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72870.97 Median E2E Latency (ms): 70495.88 P90 E2E Latency (ms): 121820.46 P99 E2E Latency (ms): 148933.09 ---------------Time to First Token---------------- Mean TTFT (ms): 2460.45 Median TTFT (ms): 1976.29 P99 TTFT (ms): 7305.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 140.57 Median TPOT (ms): 142.31 P99 TPOT (ms): 273.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 135.44 Median ITL (ms): 95.96 P95 ITL (ms): 152.93 P99 ITL (ms): 1488.37 Max ITL (ms): 6540.24 ================================================== ``` * High Concurrency ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1279.50 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169981 Request throughput (req/s): 0.25 Input token throughput (tok/s): 995.62 Output token throughput (tok/s): 132.86 Peak output token throughput (tok/s): 703.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1128.49 Concurrency: 60.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 240385.63 Median E2E Latency (ms): 236266.30 P90 E2E Latency (ms): 429882.12 P99 E2E Latency (ms): 515158.36 ---------------Time to First Token---------------- Mean TTFT (ms): 2710.44 Median TTFT (ms): 2345.63 P99 TTFT (ms): 7144.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 443.84 Median TPOT (ms): 493.29 P99 TPOT (ms): 606.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 448.23 Median ITL (ms): 296.17 P95 ITL (ms): 1869.15 P99 ITL (ms): 2708.95 Max ITL (ms): 7778.47 ================================================== ``` #### 5.2.2 Speculative Decoding Benchmark * **Model Deployment:** ```bash Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm=EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} Pending update... ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} Pending update... ``` * High Concurrency (Throughput-Optimized) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} Pending update... ``` ### 5.3 Speed Benchmark (AMD MI350X) **Test Environment:** * Hardware: AMD Instinct MI350X GPU (4x) * Model: Kimi-K2.5 (BF16) * Tensor Parallelism: 4 * SGLang Version: 0.5.9 * Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x` * ROCm: 7.0 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. :::info AMD GPU TP Constraint Kimi-K2.5 requires TP \<= 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). ::: #### 5.3.1 Latency Benchmark * **Model Deployment:** ```bash Command theme={null} SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \ sglang serve \ --model-path moonshotai/Kimi-K2.5 \ --tp 4 \ --mem-fraction-static 0.8 \ --trust-remote-code \ --reasoning-parser kimi_k2 \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 155.81 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4222 Request throughput (req/s): 0.06 Input token throughput (tok/s): 39.16 Output token throughput (tok/s): 27.09 Peak output token throughput (tok/s): 29.00 Peak concurrent requests: 2 Total token throughput (tok/s): 66.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15576.22 Median E2E Latency (ms): 12539.80 P90 E2E Latency (ms): 28150.56 P99 E2E Latency (ms): 34873.51 ---------------Time to First Token---------------- Mean TTFT (ms): 563.50 Median TTFT (ms): 594.92 P99 TTFT (ms): 830.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.61 Median TPOT (ms): 35.66 P99 TPOT (ms): 35.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.66 Median ITL (ms): 35.69 P95 ITL (ms): 35.96 P99 ITL (ms): 36.13 Max ITL (ms): 36.92 ================================================== ``` * Medium Concurrency (Balanced) ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.5 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 526.66 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40798 Request throughput (req/s): 0.15 Input token throughput (tok/s): 75.32 Output token throughput (tok/s): 77.48 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 18 Total token throughput (tok/s): 152.80 Concurrency: 14.59 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 96023.27 Median E2E Latency (ms): 93940.20 P90 E2E Latency (ms): 159449.54 P99 E2E Latency (ms): 194706.61 ---------------Time to First Token---------------- Mean TTFT (ms): 989.08 Median TTFT (ms): 886.42 P99 TTFT (ms): 1543.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 191.04 Median TPOT (ms): 195.20 P99 TPOT (ms): 238.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 186.68 Median ITL (ms): 183.82 P95 ITL (ms): 189.90 P99 ITL (ms): 673.64 Max ITL (ms): 1633.20 ================================================== ``` # Kimi-K2.6 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.6 ## 1. Model Introduction [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) is an open-source, native multimodal agentic model by Moonshot AI, delivering industry-leading coding, long-horizon execution, and agent swarm capabilities. It matches or surpasses GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro across key benchmarks. **Key Features:** * **Long-Horizon Coding**: Excels at complex, end-to-end coding tasks with 13+ hours of continuous execution and 4,000+ lines of code modification, generalizing across languages (Rust, Go, Python) and tasks (frontend, devops, performance optimization). * **Coding-Driven Design**: Transforms prompts and visual inputs into production-ready interfaces with motion-rich elements including WebGL shaders, GSAP + Framer Motion, and Three.js 3D. * **Agent Swarms Elevated**: Scales to 300 parallel sub-agents executing 4,000 coordinated steps per run. One prompt, 100+ files. * **Proactive Agents**: Powers OpenClaw, Hermes Agent, and other autonomous frameworks for 5-day continuous operation. * **Native Multimodality**: Pre-trained on vision–language tokens with MoonViT (400M parameters) for visual understanding, cross-modal reasoning, and agentic tool use grounded in visual inputs. **Benchmarks (Open-Source SOTA):**
Benchmark Score
HLE w/ tools 54.0
SWE-Bench Pro 58.6
SWE-bench Multilingual 76.7
BrowseComp 83.2
Toolathlon 50.0
AIME 2026 96.4
GPQA-Diamond 90.5
LiveCodeBench 89.6
**Recommended Generation Parameters:** * Thinking Mode: `temperature=1.0`, `top_p=0.95` * Instant Mode: `temperature=0.6`, `top_p=0.95` **Available Models:** * **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) * **NVFP4 (4-bit quantized, NVIDIA Blackwell)**: [nvidia/Kimi-K2.6-NVFP4](https://huggingface.co/nvidia/Kimi-K2.6-NVFP4) **License:** Modified MIT for the native checkpoint. The NVIDIA NVFP4 checkpoint is governed by the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.6) and [tech blog](https://kimi.com/blog/kimi-k2-6). ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory. * **NVFP4**: Use `nvidia/Kimi-K2.6-NVFP4` with `--quantization modelopt_fp4` on NVIDIA Blackwell. Use `tp=8` on B300 and `tp=4` on GB300. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.6 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. * **DP Attention**: Enable with `--dp --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.6 supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown: ## Header - At the top left is the Auntie Anne's logo (a pretzel with a halo) - The store name "**Auntie Anne's**" is printed prominently at the top - Some text below the store name appears blurred/redacted (likely store location, address, or transaction details) ## Purchase Details - **Item**: CINNAMON SUGAR - **Quantity & Price**: 1 × 17,000 - **Item Total**: 17,000 ## Financial Summary - **SUB TOTAL**: 17,000 - **GRAND TOTAL**: 17,000 - **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah) - **CHANGE DUE**: 3,000 ## Physical Description - The receipt is printed on white thermal paper - Some information in the middle section and toward the bottom is intentionally blurred/obscured - The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter) The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash. ``` #### 4.2.2 Reasoning Output Kimi-K2.6 supports both thinking mode (default) and instant mode. **Thinking Mode (default)** — reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Instant Mode (thinking off)** — disable thinking for faster responses: ```python Example theme={null} response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print("====== Response (Instant Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion. Let me think carefully: - 9.11 means 9 + 11/100 = 9.11 - 9.9 means 9 + 9/10 = 9.90 So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11. The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9). So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79. I should explain this clearly and address the common misconception. ====== Response (Thinking Mode) ====== Mathematically, **9.9 is bigger**. Here's why: **9.9 = 9.90** When comparing decimals, you need to look at the same place values: - 9.11 = 9 ones, 1 tenth, and 1 hundredth - 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90) Since **0.90 > 0.11**, it follows that **9.9 > 9.11**. The difference is: 9.9 - 9.11 = 0.79 **Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger. ====== Response (Instant Mode) ====== I need to compare 9.11 and 9.9. Let me think carefully by aligning the decimal places: - 9.11 = 9 and 11/100 = 9.11 - 9.9 = 9 and 9/10 = 9.90 Since 0.90 > 0.11 **9.9 is bigger.** This is a common trick question because people sometimes mistakenly compare 11 and 9 as whole numbers after the decimal point, forgetting that 9.9 = 9.90, which is greater than 9.11. ``` #### 4.2.3 Tool Calling Kimi-K2.6 supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently **22°C and sunny**. ☀️ It's a nice, warm day there—great for being outdoors! ``` #### 4.2.4 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain. I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items. === Content === Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR). Let me search for this product and similar items for you. === Tool Calls === Function: search_product Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.5 Speculative Decoding **NVIDIA** Deploy Kimi-K2.6 with the following command (H200/B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` Deploy Kimi-K2.6-NVFP4 with the following command (B300, all features enabled): ```shell Command theme={null} sglang serve \ --model-path nvidia/Kimi-K2.6-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: 8× NVIDIA H200 * Model: moonshotai/Kimi-K2.6 (INT4) * Tensor Parallelism: 8 * SGLang version: 0.5.9 * Reasoning Parser: `kimi_k2` * Tool Call Parser: `kimi_k2` #### NVIDIA NVFP4 Accuracy Reference NVIDIA reports the following accuracy results for `nvidia/Kimi-K2.6-NVFP4` compared with the native INT4 baseline, using `temperature=1.0`, `top_p=0.95`, and max tokens 128,000:
Precision GPQA Diamond SciCode τ²-Bench Telecom MMMU Pro AA-LCR IFBench
Baseline (INT4) 90.9 52.6 98.2 75.6 71.0 73.9
NVFP4 90.4 54.4 98.0 76.5 71.8 73.9
#### 5.1.1 K2-Vendor-Verifier (Tool Calling) * Dataset: [K2-Vendor-Verifier](https://github.com/MoonshotAI/K2-Vendor-Verifier) tool-calls dataset (2,000 requests) * Evaluation Tool: K2-Vendor-Verifier `tool_calls_eval.py` * Settings: temperature=1.0, max\_tokens=64,000, concurrency=256 **Evaluation Command:** ```shell Command theme={null} cd K2-Vendor-Verifier python tool_calls_eval.py tool-calls/samples.jsonl \ --model "moonshotai/Kimi-K2.6" \ --base-url "http://localhost:30000/v1" \ --api-key "placeholder" \ --concurrency 256 \ --temperature 1.0 \ --max-tokens 64000 \ --output kimi-k26-results.jsonl ``` **Results:**
Metric Value
Success Rate 99.95% (1999/2000)
Tool Call Triggered 970
Tool Call Valid 89.6% (869/970)
Tool Call Invalid (schema error) 10.4% (101/970)
#### 5.1.2 AIME 2025 * Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems) * Evaluation Tool: [NVIDIA NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills) * Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers) * Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 32 seeds **Evaluation Command:** ```shell Command theme={null} # Prepare dataset python3 nemo_skills/dataset/aime25/prepare.py # Run 32 seeds in parallel for RS in $(seq 0 31); do python3 nemo_skills/inference/generate.py \ input_file=nemo_skills/dataset/aime25/test.jsonl \ output_file=results/kimi-k26/aime25/output-rs${RS}.jsonl \ prompt_config=eval/matharena/aime \ prompt_format=openai \ +server.server_type=openai \ +server.model=moonshotai/Kimi-K2.6 \ +server.base_url=http://localhost:30000/v1 \ ++inference.temperature=1.0 \ ++inference.top_p=0.95 \ ++inference.tokens_to_generate=131072 \ ++inference.random_seed=${RS} \ max_concurrent_requests=512 & done ``` **Results:**
Evaluation Mode Accuracy
pass\@1 (avg-of-32) 98.9% (29.7/30)
majority\@32 100.0% (30/30)
pass\@32 100.0%
> 22 out of 32 seeds achieved a perfect score of 30/30. The remaining 10 seeds each missed exactly 1 problem (29/30). #### 5.1.3 GPQA Diamond * Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions, 4-choice multiple choice) * Evaluation Tool: [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) with `inspect_evals/gpqa_diamond` * Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 4 epochs, cot=True **Evaluation Command:** ```shell Command theme={null} OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ inspect eval inspect_evals/gpqa_diamond \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 131072 \ --temperature 1.0 \ --top-p 0.95 \ --max-connections 128 \ -T cot=True ``` **Results (partial — 553/792 samples across 4 epochs):**
Evaluation Mode Accuracy
pass\@1 (avg across epochs) 96.9%
Epoch Accuracy
1 96.4% (160/166)
2 96.9% (156/161)
3 96.9% (155/160)
4 98.5% (65/66)
#### 5.1.4 OCRBench * Dataset: [OCRBench](https://huggingface.co/datasets/echo840/OCRBench) (1,000 questions with images) * Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based) * Settings: max\_tokens=4,096, thinking mode enabled (opensource) **Evaluation Command:** ```shell Command theme={null} cd Kimi-Vendor-Verifier OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ python3 eval.py ocrbench \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 4096 \ --think-mode opensource \ --thinking \ --max-connections 256 ``` **Results:**
Evaluation Mode Accuracy
pass\@1 90.8%
#### 5.1.5 MMMU Pro Vision * Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images) * Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based) * Settings: max\_tokens=32,768, thinking mode (default), max\_connections=256 > **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher. **Evaluation Command:** ```shell Command theme={null} cd Kimi-Vendor-Verifier OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \ python3 eval.py mmmu \ --model openai/moonshotai/Kimi-K2.6 \ --max-tokens 32768 \ --think-mode none \ --max-connections 256 ``` **Results (1,481/1,730 samples completed):**
Evaluation Mode Accuracy
pass\@1 82.2%
### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (8x) * Model: Kimi-K2.6 * Tensor Parallelism: 8 * SGLang Version: 0.5.9 Kimi-K2.6 shares the same architecture as K2.5. Speed benchmarks are expected to be equivalent. The results below are measured with K2.5 and serve as a reference. We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. #### 5.2.1 Latency Benchmark * **Model Deployment:** ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` **Scenario 1: Chat (1K/1K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 39.77 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4221 Request throughput (req/s): 0.25 Input token throughput (tok/s): 153.40 Output token throughput (tok/s): 106.10 Peak output token throughput (tok/s): 156.00 Peak concurrent requests: 2 Total token throughput (tok/s): 259.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3972.87 Median E2E Latency (ms): 4044.55 P90 E2E Latency (ms): 7046.30 P99 E2E Latency (ms): 7441.13 ---------------Time to First Token---------------- Mean TTFT (ms): 176.89 Median TTFT (ms): 154.24 P99 TTFT (ms): 285.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.22 Median TPOT (ms): 9.32 P99 TPOT (ms): 12.72 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.02 Median ITL (ms): 8.80 P95 ITL (ms): 13.23 P99 ITL (ms): 14.17 Max ITL (ms): 29.38 ================================================== ``` * Medium Concurrency (Balanced) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 158.05 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40775 Request throughput (req/s): 0.51 Input token throughput (tok/s): 250.99 Output token throughput (tok/s): 258.18 Peak output token throughput (tok/s): 1103.00 Peak concurrent requests: 19 Total token throughput (tok/s): 509.17 Concurrency: 14.09 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27837.05 Median E2E Latency (ms): 23508.00 P90 E2E Latency (ms): 57126.31 P99 E2E Latency (ms): 66044.35 ---------------Time to First Token---------------- Mean TTFT (ms): 374.30 Median TTFT (ms): 375.51 P99 TTFT (ms): 695.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 53.25 Median TPOT (ms): 57.93 P99 TPOT (ms): 85.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 53.95 Median ITL (ms): 53.97 P95 ITL (ms): 84.74 P99 ITL (ms): 244.84 Max ITL (ms): 655.61 ================================================== ``` * High Concurrency (Throughput-Optimized) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 996.64 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252588 Request throughput (req/s): 0.50 Input token throughput (tok/s): 250.67 Output token throughput (tok/s): 253.51 Peak output token throughput (tok/s): 1199.00 Peak concurrent requests: 104 Total token throughput (tok/s): 504.18 Concurrency: 92.70 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 184773.75 Median E2E Latency (ms): 174183.65 P90 E2E Latency (ms): 343625.28 P99 E2E Latency (ms): 404284.53 ---------------Time to First Token---------------- Mean TTFT (ms): 1289.59 Median TTFT (ms): 1313.35 P99 TTFT (ms): 2346.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 364.70 Median TPOT (ms): 403.32 P99 TPOT (ms): 452.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 363.82 Median ITL (ms): 316.21 P95 ITL (ms): 745.91 P99 ITL (ms): 1345.88 Max ITL (ms): 3118.59 ================================================== ``` **Scenario 2: Reasoning (1K/8K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 680.26 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44455 Request throughput (req/s): 0.01 Input token throughput (tok/s): 8.97 Output token throughput (tok/s): 65.36 Peak output token throughput (tok/s): 151.00 Peak concurrent requests: 2 Total token throughput (tok/s): 74.33 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 68019.29 Median E2E Latency (ms): 70568.85 P90 E2E Latency (ms): 113237.40 P99 E2E Latency (ms): 121682.34 ---------------Time to First Token---------------- Mean TTFT (ms): 206.17 Median TTFT (ms): 177.28 P99 TTFT (ms): 445.37 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.36 Median TPOT (ms): 15.89 P99 TPOT (ms): 16.43 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.26 Median ITL (ms): 15.85 P95 ITL (ms): 17.50 P99 ITL (ms): 23.21 Max ITL (ms): 45.22 ================================================== ``` * Medium Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 2475.98 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 318166 Request throughput (req/s): 0.03 Input token throughput (tok/s): 16.02 Output token throughput (tok/s): 128.56 Peak output token throughput (tok/s): 847.00 Peak concurrent requests: 18 Total token throughput (tok/s): 144.58 Concurrency: 14.62 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 452592.46 Median E2E Latency (ms): 486002.05 P90 E2E Latency (ms): 833197.57 P99 E2E Latency (ms): 957399.48 ---------------Time to First Token---------------- Mean TTFT (ms): 359.38 Median TTFT (ms): 350.78 P99 TTFT (ms): 500.36 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 111.18 Median TPOT (ms): 122.76 P99 TPOT (ms): 145.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 113.69 Median ITL (ms): 122.81 P95 ITL (ms): 147.87 P99 ITL (ms): 151.03 Max ITL (ms): 272.05 ================================================== ``` **Scenario 3: Summarization (8K/1K)** * Low Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 120.73 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.08 Input token throughput (tok/s): 347.41 Output token throughput (tok/s): 34.96 Peak output token throughput (tok/s): 73.00 Peak concurrent requests: 2 Total token throughput (tok/s): 382.36 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12068.56 Median E2E Latency (ms): 10211.36 P90 E2E Latency (ms): 23203.32 P99 E2E Latency (ms): 30677.66 ---------------Time to First Token---------------- Mean TTFT (ms): 1625.64 Median TTFT (ms): 1526.63 P99 TTFT (ms): 3743.51 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.95 Median TPOT (ms): 23.95 P99 TPOT (ms): 35.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.80 Median ITL (ms): 21.73 P95 ITL (ms): 59.56 P99 ITL (ms): 61.10 Max ITL (ms): 62.70 ================================================== ``` * Medium Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 389.96 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41670 Request throughput (req/s): 0.21 Input token throughput (tok/s): 769.36 Output token throughput (tok/s): 106.86 Peak output token throughput (tok/s): 304.00 Peak concurrent requests: 19 Total token throughput (tok/s): 876.22 Concurrency: 14.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72870.97 Median E2E Latency (ms): 70495.88 P90 E2E Latency (ms): 121820.46 P99 E2E Latency (ms): 148933.09 ---------------Time to First Token---------------- Mean TTFT (ms): 2460.45 Median TTFT (ms): 1976.29 P99 TTFT (ms): 7305.53 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 140.57 Median TPOT (ms): 142.31 P99 TPOT (ms): 273.40 ---------------Inter-Token Latency---------------- Mean ITL (ms): 135.44 Median ITL (ms): 95.96 P95 ITL (ms): 152.93 P99 ITL (ms): 1488.37 Max ITL (ms): 6540.24 ================================================== ``` * High Concurrency ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 1279.50 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169981 Request throughput (req/s): 0.25 Input token throughput (tok/s): 995.62 Output token throughput (tok/s): 132.86 Peak output token throughput (tok/s): 703.00 Peak concurrent requests: 67 Total token throughput (tok/s): 1128.49 Concurrency: 60.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 240385.63 Median E2E Latency (ms): 236266.30 P90 E2E Latency (ms): 429882.12 P99 E2E Latency (ms): 515158.36 ---------------Time to First Token---------------- Mean TTFT (ms): 2710.44 Median TTFT (ms): 2345.63 P99 TTFT (ms): 7144.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 443.84 Median TPOT (ms): 493.29 P99 TPOT (ms): 606.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 448.23 Median ITL (ms): 296.17 P95 ITL (ms): 1869.15 P99 ITL (ms): 2708.95 Max ITL (ms): 7778.47 ================================================== ``` ### 5.3 Speed Benchmark (AMD MI350X) **Test Environment:** * Hardware: AMD Instinct MI350X GPU (4x) * Model: Kimi-K2.6 (INT4) * Tensor Parallelism: 4 * SGLang Version: 0.5.9 * Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x` * ROCm: 7.0 We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation. **AMD GPU TP Constraint**: Kimi-K2.6 requires TP ≤ 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). #### 5.3.1 Latency Benchmark * **Model Deployment:** ```shell Command theme={null} SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \ sglang serve \ --model-path moonshotai/Kimi-K2.6 \ --tp 4 \ --mem-fraction-static 0.8 \ --trust-remote-code \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --kv-cache-dtype fp8_e4m3 \ --host 0.0.0.0 \ --port 30000 ``` * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * **Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 155.81 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4222 Request throughput (req/s): 0.06 Input token throughput (tok/s): 39.16 Output token throughput (tok/s): 27.09 Peak output token throughput (tok/s): 29.00 Peak concurrent requests: 2 Total token throughput (tok/s): 66.24 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15576.22 Median E2E Latency (ms): 12539.80 P90 E2E Latency (ms): 28150.56 P99 E2E Latency (ms): 34873.51 ---------------Time to First Token---------------- Mean TTFT (ms): 563.50 Median TTFT (ms): 594.92 P99 TTFT (ms): 830.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 35.61 Median TPOT (ms): 35.66 P99 TPOT (ms): 35.77 ---------------Inter-Token Latency---------------- Mean ITL (ms): 35.66 Median ITL (ms): 35.69 P95 ITL (ms): 35.96 P99 ITL (ms): 36.13 Max ITL (ms): 36.92 ================================================== ``` * Medium Concurrency (Balanced) ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-K2.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 526.66 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40798 Request throughput (req/s): 0.15 Input token throughput (tok/s): 75.32 Output token throughput (tok/s): 77.48 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 18 Total token throughput (tok/s): 152.80 Concurrency: 14.59 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 96023.27 Median E2E Latency (ms): 93940.20 P90 E2E Latency (ms): 159449.54 P99 E2E Latency (ms): 194706.61 ---------------Time to First Token---------------- Mean TTFT (ms): 989.08 Median TTFT (ms): 886.42 P99 TTFT (ms): 1543.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 191.04 Median TPOT (ms): 195.20 P99 TPOT (ms): 238.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 186.68 Median ITL (ms): 183.82 P95 ITL (ms): 189.90 P99 ITL (ms): 673.64 Max ITL (ms): 1633.20 ================================================== ``` # Kimi-K2.7-Code Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input. ## 1. Model Introduction [Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model by Moonshot AI, built on top of Kimi-K2.6. It improves real-world long-horizon coding task completion while reducing thinking-token usage by approximately 30% compared with Kimi-K2.6. **Key Features:** * **Coding-Focused Agentic Model**: Optimized for end-to-end coding workflows and complex software engineering tasks. * **Token Efficiency**: Reduces thinking-token usage by approximately 30% versus Kimi-K2.6. * **K2.6-Compatible Deployment**: Shares the same architecture as Kimi-K2.5/Kimi-K2.6, so the SGLang deployment method can be reused with the new model ID. * **Native Multimodality**: Shares Kimi-K2.6's native multimodal architecture with a MoonViT vision encoder (400M parameters) and supports image and video (experimental) input. **Benchmarks:**
Benchmark Kimi-K2.6 Kimi-K2.7-Code
Kimi Code Bench v2 50.9 62.0
Program Bench 48.3 53.6
MLS Bench Lite 26.7 35.1
Kimi Claw 24/7 Bench 42.9 46.9
MCP Atlas 69.4 76.0
MCP Mark Verified 72.8 81.1
**Recommended Generation Parameters:** * Thinking Mode: `temperature=1.0`, `top_p=0.95` * Kimi-K2.7-Code forces thinking and preserve-thinking behavior; instant mode is not supported. **Available Models:** * **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) **License:** Modified MIT for the native checkpoint. For details, see the [official model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code). ## 2. SGLang Installation Refer to the [official SGLang installation guide](/docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory. * **Context Length**: The model supports a 256K context length. Use a shorter `--context-length` when you need to reserve memory for larger batches. * **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`. * **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. * **DP Attention**: Enable with `--dp --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required. * **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs. * **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls. * **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](/docs/basic_usage/send_request). ### 4.2 Advanced Usage #### 4.2.1 Multimodal (Vision + Text) Input Kimi-K2.7-Code supports native multimodal input with images: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "What is in this image? Describe it in detail." } ] } ] ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown: ## Header - At the top left is the Auntie Anne's logo (a pretzel with a halo) - The store name "**Auntie Anne's**" is printed prominently at the top - Some text below the store name appears blurred/redacted (likely store location, address, or transaction details) ## Purchase Details - **Item**: CINNAMON SUGAR - **Quantity & Price**: 1 × 17,000 - **Item Total**: 17,000 ## Financial Summary - **SUB TOTAL**: 17,000 - **GRAND TOTAL**: 17,000 - **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah) - **CHANGE DUE**: 3,000 ## Physical Description - The receipt is printed on white thermal paper - Some information in the middle section and toward the bottom is intentionally blurred/obscured - The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter) The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash. ``` #### 4.2.2 Reasoning Output Kimi-K2.7-Code forces thinking mode and preserve-thinking behavior. **Thinking Mode (default)** — reasoning content is automatically separated: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning Content (Thinking Mode) ======") print(response.choices[0].message.reasoning_content) print("====== Response (Thinking Mode) ======") print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content (Thinking Mode) ====== The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion. Let me think carefully: - 9.11 means 9 + 11/100 = 9.11 - 9.9 means 9 + 9/10 = 9.90 So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11. The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9). So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79. I should explain this clearly and address the common misconception. ====== Response (Thinking Mode) ====== Mathematically, **9.9 is bigger**. Here's why: **9.9 = 9.90** When comparing decimals, you need to look at the same place values: - 9.11 = 9 ones, 1 tenth, and 1 hundredth - 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90) Since **0.90 > 0.11**, it follows that **9.9 > 9.11**. The difference is: 9.9 - 9.11 = 0.79 **Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger. ``` #### 4.2.3 Preserve Thinking Kimi-K2.7-Code keeps reasoning content across multi-turn interactions. This behavior is enabled by default and cannot be disabled. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) messages = [ { "role": "user", "content": "Tell me three random numbers." }, { "role": "assistant", "reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.", "content": "473, 921, 235" }, { "role": "user", "content": "What are the other two numbers you have in mind?" } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=messages, stream=False, max_tokens=4096, ) print(response.choices[0].message.content) ``` Some OpenAI-compatible deployments use `reasoning` instead of `reasoning_content` in assistant messages. Use the field your serving stack exposes. #### 4.2.4 Tool Calling Kimi-K2.7-Code supports tool calling capabilities for agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'tool_calls') and delta.tool_calls: for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = {'name': None, 'arguments': ''} if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments if delta.content: print(delta.content, end="", flush=True) for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") ``` **Output Example:** ```text Output theme={null} Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": "The weather in Beijing is 22°C and sunny." } ] final_response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=messages ) print(final_response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The weather in Beijing is currently **22°C and sunny**. ☀️ It's a nice, warm day there—great for being outdoors! ``` #### 4.2.5 Multimodal + Tool Calling (Agentic Vision) Combine vision understanding with tool calling for advanced agentic tasks: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "search_product", "description": "Search for a product by name or description", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The product name or description to search for" } }, "required": ["query"] } } } ] response = client.chat.completions.create( model="moonshotai/Kimi-K2.7-Code", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Can you identify this product and search for similar items?" } ] } ], tools=tools ) msg = response.choices[0].message # Print reasoning process if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) # Print response content if msg.content: print("=== Content ===") print(msg.content) # Print tool calls if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} === Reasoning === The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain. I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items. === Content === Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR). Let me search for this product and similar items for you. === Tool Calls === Function: search_product Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"} ``` #### 4.2.6 Deployment Command Example Deploy Kimi-K2.7-Code with the following command (H200/B300, reasoning and tool parsing enabled): ```shell Command theme={null} sglang serve \ --model-path moonshotai/Kimi-K2.7-Code \ --tp 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` For GB300, use `--tp 4`. ## 5. Benchmark The following results are from the official Kimi-K2.7-Code model card. They were evaluated with thinking mode enabled through Kimi Code CLI at `temperature=1.0`, `top_p=0.95`, and a 262,144-token context length unless otherwise stated.
Category Benchmark Kimi-K2.6 Kimi-K2.7-Code
Coding Kimi Code Bench v2 50.9 62.0
Coding Program Bench 48.3 53.6
Coding MLS Bench Lite 26.7 35.1
Agentic Kimi Claw 24/7 Bench 42.9 46.9
Agentic MCP Atlas 69.4 76.0
Agentic MCP Mark Verified 72.8 81.1
# Kimi-K3 Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3 Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA, AMD, and NPU recipes. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest # NVIDIA (CUDA) docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260817 # AMD MI350X / MI355X (ROCm) ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. ```bash Command theme={null} docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 ``` For host and platform setup, see the [NPU installation guide](../../../docs/hardware-platforms/ascend-npus/getting-started/installation) and the [quick start guide](../../../docs/hardware-platforms/ascend-npus/getting-started/quick_start). **Weights (NPU):** [Kimi-K3-W4A8](https://www.modelscope.cn/models/sgl-npu/Kimi-K3-W4A8) (W4A8, 1.49 TB) · [Kimi-K3-DSpark](https://www.modelscope.cn/models/RadixArk/Kimi-K3-DSpark) (DSPARK draft, 4.5 GB) Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Atlas 800I A3 4×8 — 32 cards / 64 dies), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image. **PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)). **Strategy** — the operating point within that shape: * **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP. * **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x. * **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`). `Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`. **Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows. `--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Atlas 800I A3 uses `--max-mamba-cache-size` instead.) ### Mamba ratio calculator `--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio: ```text theme={null} ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes)) ``` * `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two. * `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`. * `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype. * `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (\~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8. * `L` — average total request length in tokens: input + output. ## Advanced Features Playground The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput. K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`). Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights are scheduled to release by July 27, 2026**. The recipes on this page were validated on the public [`sgl-project/sglang` `kimi-k3` branch](https://github.com/sgl-project/sglang/tree/kimi-k3) — the HuggingFace repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be available at launch. Every cell in the Deploy panel above is currently marked **Final Verification In Progress**: the recipe runs, but its serving round on the final weights and current code is still open. Re-measure throughput and accuracy before you rely on any of them. **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code). **Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart). ## 2. Configuration Tips **Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Atlas 800I A3: `--max-mamba-cache-size`, no calculator.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side). Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload: | Lever | Effect | | ------------------------------------------------ | ----------------------------------------------------------------------------------------------- | | `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 | | `--mamba-ssm-dtype bfloat16` | \~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton | | `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect | | `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` | | `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) | Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). **MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected with the pinned FlashInfer 0.6.17 dependency; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on. The published Docker images install the matching official `flashinfer-python`, `flashinfer-cubin`, and `flashinfer-jit-cache` packages. **Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode. **Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV. **DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting. **Per-platform notes:** | Platform | Topology | Notes | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced | | GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected | | B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy | | GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | | H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | | H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | | MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention (`SGLANG_MLA_DECODE_TUNE=1` for gfx950 MLA decode geometry), graph bs up to 256, fp8 kvcache; DSPARK supported | | Atlas 800I A3 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | **DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: * DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it. * Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300). * No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness). * Explicit `tokenspeed_mla` force-rewrites `--kv-cache-dtype` to fp8; the default `cutedsl_mla` serves either dtype. * Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context. * Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `--moe-a2a-backend` is set. No cell has a serving round in this exact shape — treat them as starting points to verify. ## 3. Advanced Usage ### 3.1 Reasoning K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="moonshotai/Kimi-K3", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="high", # "low" | "high" | "max" (default max) ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Pending update... ``` ### 3.2 Tool Calling Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Atlas 800I A3.) ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="moonshotai/Kimi-K3", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Pending update... ``` ### 3.3 HiCache (Hierarchical KV Caching) K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads. * On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`. * Low-Latency and the Hopper recipes take all tiers unchanged. ### 3.4 PD Disaggregation PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state. * **Transfer**: the cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground. * **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers. * **Decode state pool**: chunk cache — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above. #### Deep PP for prefill Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory. * Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP. * Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency \~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there. * DSPARK off (`pp_size == 1` required) — on B200/GB200 that applies to `Default` as well. * Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side. ```bash Command theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30100 \ --host 0.0.0.0 --port 8000 \ --disable-circuit-breaker \ --health-check-interval-secs 999999 ``` Clients then send requests to the router (`:8000`) instead of an individual role server. ### 3.5 VLM Serving Profiles The open-source K3 serving contract currently supports **image input only** — its processor rejects video and audio input. #### VLM feature transport Use **VLM Transport** in the command picker. `Auto` is a topology-aware starting point, not a claim that one configuration is fastest for every workload. | Picker selection | Processor-to-scheduler feature path | | ------------------------------- | ---------------------------------------------- | | Auto · single-node Unified CUDA | CUDA IPC | | Auto · Unified GB200/GB300 | CUDA VMM when IMEX is available; CPU otherwise | | Auto · PD or other topologies | CPU | | CPU | CPU, with no GPU feature pool | CUDA IPC and CUDA VMM reserve up to `SGLANG_MM_FEATURE_CACHE_MB` (1 GiB by default) on the base GPU and fall back to CPU per tensor when full. This setting does not control EPD encoder output or PD KV/KDA transfer. K3 already defaults to 2 processor workers and 16 I/O workers; leave those flags unset unless tuning. #### VLM compatibility | Feature | K3 behavior | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). | | EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). | | MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. | | MM feature transport | Processor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends. | | ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. | #### Should ViT BCG be enabled? Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM. * The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving. * Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes. * The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring. #### Low-HBM VLM Use this profile when keeping HBM headroom matters more than peak concurrency. It removes the GPU feature pool, keeps ViT BCG disabled, halves the context window, caps concurrency, and lowers the static-memory target: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 8 \ --context-length 65536 \ --enable-symm-mem \ --mem-fraction-static 0.82 \ --mm-feature-transport cpu \ --reasoning-parser kimi_k3 \ --tool-call-parser kimi_k3 \ --host 0.0.0.0 \ --port 30000 ``` `--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated. ### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell) **The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP). Two presets come out of this, at `N = 8k` GPUs: | Preset | What it trades | Pick it for | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------ | | **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. | | **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, \~1.8× ITL. | Context ≥ \~16K, or per-replica concurrency past 128. | * **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5. * The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's \~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape. The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`: ```bash Command theme={null} SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \ sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 32 --ep-size 32 \ --enable-dp-attention --dp-size 4 --enable-dp-lm-head \ --nnodes 4 --node-rank --dist-init-addr :20000 \ --moe-a2a-backend megamoe --moe-runner-backend deep_gemm \ --kv-cache-dtype fp8_e4m3 \ --mamba-ssm-dtype bfloat16 \ --mamba-radix-cache-strategy extra_buffer_lazy \ --mem-fraction-static 0.92 \ --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \ --host 0.0.0.0 --port 30000 ``` Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation: | GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` | | ---- | --------------- | ----------------- | ------------------------- | ----------- | | 16 | 2×8 | 4×4 | 16 | 2 | | 32 | 4×8 | 8×4 | 32 | 4 | | 64 | 8×8 | 16×4 | 64 | 8 | For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator). Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing. Decisions the preset already makes: * **MegaMoE + `deep_gemm`** — the fused DeepGEMM all-to-all/MoE path used by these large-scale DP/EP throughput presets, with K3's SiTU activation. * **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not. * **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales. No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet. # Kimi-Linear Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-Linear ## AMD GPU Support ## 1. Model Introduction Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory. This generation delivers comprehensive upgrades across the board: Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating. Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention. Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons. High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT). For more details, please refer to the \[official Kimi Linear GitHub Repository]: [https://github.com/MoonshotAI/Kimi-Linear](https://github.com/MoonshotAI/Kimi-Linear) ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Launch the docker ```shell Command theme={null} docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x ``` ```shell Command theme={null} docker run -d -it --ipc=host --network=host --privileged \ --cap-add=CAP_SYS_ADMIN \ --device=/dev/kfd --device=/dev/dri --device=/dev/mem \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v /:/work \ -e SHELL=/bin/bash \ --name Kimi-linear \ lmsysorg/sglang:v0.5.7-rocm700-mi30x \ /bin/bash ``` #### 4.2.2 pre-installation steps inside the docker ```shell Command theme={null} pip install sentencepiece tiktoken ``` #### 4.2.3 Launch the server ```shell Command theme={null} export SGLANG_ROCM_FUSED_DECODE_MLA=0 SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` ## 5. Benchmark ### 5.1 Speed Benchmark Test Environment: Hardware: AMD MI300X GPU Model: Kimi-Linear-48B-A3B-Instruct Tensor Parallelism: 4 sglang version: 0.5.7 * **Model Deployment** ```bash Command theme={null} SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` ### 5.1.1 Low Concurrency (Latency-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 23.86 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4220 Total generated tokens (retokenized): 4001 Request throughput (req/s): 0.42 Input token throughput (tok/s): 255.70 Output token throughput (tok/s): 176.86 Peak output token throughput (tok/s): 190.00 Peak concurrent requests: 2 Total token throughput (tok/s): 432.56 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2383.93 Median E2E Latency (ms): 1911.63 ---------------Time to First Token---------------- Mean TTFT (ms): 141.33 Median TTFT (ms): 126.27 P99 TTFT (ms): 294.76 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.32 Median TPOT (ms): 5.33 P99 TPOT (ms): 5.36 ---------------Inter-Token Latency---------------- Mean ITL (ms): 5.33 Median ITL (ms): 5.32 P95 ITL (ms): 5.44 P99 ITL (ms): 5.58 Max ITL (ms): 11.46 ================================================== ``` ### 5.1.2 Medium Concurrency (Balanced) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 31.38 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40805 Total generated tokens (retokenized): 39667 Request throughput (req/s): 2.55 Input token throughput (tok/s): 1264.13 Output token throughput (tok/s): 1300.37 Peak output token throughput (tok/s): 1801.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2564.50 Concurrency: 14.13 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5543.18 Median E2E Latency (ms): 5755.31 ---------------Time to First Token---------------- Mean TTFT (ms): 175.25 Median TTFT (ms): 137.87 P99 TTFT (ms): 292.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.75 Median TPOT (ms): 10.87 P99 TPOT (ms): 16.74 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.54 Median ITL (ms): 7.95 P95 ITL (ms): 13.68 P99 ITL (ms): 116.80 Max ITL (ms): 299.89 ================================================== ``` ### 5.1.3 High Concurrency (Throughput-Optimized) * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model moonshotai/Kimi-Linear-48B-A3B-Instruct \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 \ --request-rate inf ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 79.71 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252662 Total generated tokens (retokenized): 228448 Request throughput (req/s): 6.27 Input token throughput (tok/s): 3134.20 Output token throughput (tok/s): 3169.72 Peak output token throughput (tok/s): 6109.00 Peak concurrent requests: 110 Total token throughput (tok/s): 6303.92 Concurrency: 94.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 15113.92 Median E2E Latency (ms): 13851.52 ---------------Time to First Token---------------- Mean TTFT (ms): 564.46 Median TTFT (ms): 226.04 P99 TTFT (ms): 2683.14 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.63 Median TPOT (ms): 31.28 P99 TPOT (ms): 38.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.85 Median ITL (ms): 16.29 P95 ITL (ms): 123.42 P99 ITL (ms): 157.80 Max ITL (ms): 2481.11 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * Server Command ```shell Command theme={null} SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tokenizer-path moonshotai/Kimi-Linear-48B-A3B-Instruct \ --tp 4 \ --trust-remote-code ``` * Benchmark Command ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * **Result**: ```text Output theme={null} Accuracy: 0.705 Invalid: 0.000 Latency: 11.855 s Output throughput: 3224.982 token/s ``` # Nemotron3-Nano Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano ## 1. Model Introduction `NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks. The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported. At a high level: * **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only. * **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth. * **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs. ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install nightly wheel through: ```bash Command theme={null} uv pip install --prerelease=allow sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/ ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend**: **H200**: Use flash attention 3 backend by default. **B200**: Use flashinfer backend by default. * **TP support**: To set tp size, use `--tp <1|2|4|8>`. * **FP8 KV cache**: To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`. ## 4. Model Invocation ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Summarize what MoE models are in 5 bullets."}, ], temperature=0.7, max_tokens=256, ) print(resp.choices[0].message.content) ``` Streaming chat completion ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` ### 4.2 Reasoning To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable\_thinking to False, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=0.7, max_tokens=512, ) print(resp.choices[0].message.reasoning_content) # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=0.6, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(resp.choices[0].message.reasoning_content) ``` ### 4.3 Tool calling To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool\_calls. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": { "type": "integer", "description": "The total amount of the bill" }, "tip_percentage": { "type": "integer", "description": "The percentage of tip to be applied" } }, "required": ["bill_total", "tip_percentage"] } } } ] completion = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", messages=[ {"role": "system", "content": ""}, {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU **FP8 variant** * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --trust-remote-code \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 183.18 Total input tokens: 2081726 Total input text tokens: 2081726 Total input vision tokens: 0 Total generated tokens: 2116125 Total generated tokens (retokenized): 1076256 Request throughput (req/s): 22.36 Input token throughput (tok/s): 11364.25 Output token throughput (tok/s): 11552.04 Peak output token throughput (tok/s): 24692.00 Peak concurrent requests: 294 Total token throughput (tok/s): 22916.30 Concurrency: 251.19 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 11233.74 Median E2E Latency (ms): 11142.97 ---------------Time to First Token---------------- Mean TTFT (ms): 172.99 Median TTFT (ms): 116.57 P99 TTFT (ms): 1193.68 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 21.74 Median TPOT (ms): 21.14 P99 TPOT (ms): 41.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 21.45 Median ITL (ms): 9.06 P95 ITL (ms): 62.59 P99 ITL (ms): 110.83 Max ITL (ms): 5368.19 ================================================== ``` **BF16 variant** * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 360.22 Total input tokens: 2081726 Total input text tokens: 2081726 Total input vision tokens: 0 Total generated tokens: 2087288 Total generated tokens (retokenized): 1940652 Request throughput (req/s): 11.37 Input token throughput (tok/s): 5779.10 Output token throughput (tok/s): 5794.55 Peak output token throughput (tok/s): 9169.00 Peak concurrent requests: 276 Total token throughput (tok/s): 11573.65 Concurrency: 249.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 21965.10 Median E2E Latency (ms): 21706.35 ---------------Time to First Token---------------- Mean TTFT (ms): 211.54 Median TTFT (ms): 93.06 P99 TTFT (ms): 2637.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 43.27 Median TPOT (ms): 43.04 P99 TPOT (ms): 61.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 42.77 Median ITL (ms): 28.46 P95 ITL (ms): 71.85 P99 ITL (ms): 113.20 Max ITL (ms): 5237.28 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: NVIDIA B200 GPU * Model: BF16 checkpoint **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \ --trust-remote-code \ --reasoning-parser nemotron_3 ``` **Run Benchmark with lm-eval** ```bash Command theme={null} pip install lm-eval[api]==0.4.9.2 lm_eval --model local-completions --tasks gsm8k --model_args "model=nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=4,max_retries=3,tokenized_requests=False,max_lengths=16384" --gen_kwargs '{"chat_template_kwargs":{"thinking":true}}' --batch_size 256 ``` **Test Results:** ```text Output theme={null} |Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr| |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:| |gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.5603|± |0.0137| | | |strict-match | 5|exact_match|↑ |0.8453|± |0.0100| ``` # Nemotron 3 Nano Omni Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni ## 1. Model Introduction `NVIDIA Nemotron 3 Nano Omni` is a 30B-parameter hybrid MoE multimodal model that activates only 3B parameters per forward pass, combining vision and audio encoders into a unified architecture. Part of the Nemotron 3 family, it is designed to power multimodal sub-agents that perceive and reason across vision, audio, and language in a single inference loop — eliminating the fragmented stacks of separate models for each modality. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture for efficient routing and sequence modeling. * **30B total / 3B active parameters:** Delivers strong multimodal accuracy at a fraction of the cost of dense models. * **1M token context window:** Sustains coherent agent state across extended multimodal workflows — screen history, document content, and audio context remain in view without re-ingestion. * **Unified vision and audio encoders:** One model replaces fragmented multimodal stacks; vision and audio perception happen in the same forward pass. * **3D Convolution (Conv3D):** Efficient temporal-spatial processing for video inputs. * **Efficient Video Sampling (EVS):** Enables longer video processing at the same compute budget via temporal-aware perception and adaptive frame sampling. * **FP8 and NVFP4 quantization:** FP8 supports deployment from workstation (RTX 6000, DGX Spark) to cloud (H100, H200, B200, A100, L40S); NVFP4 requires Blackwell hardware. * **9x higher throughput** than other open omni models at the same interactivity level. * **\~20% higher multimodal intelligence** compared to the best open alternative. * **Post-trained with multi-environment reinforcement learning** via NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, improving instruction following and convergence to correct multimodal answers. **Modalities:** Input: text, image, video, audio — Output: text **Supported GPUs:** NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000 Available model variants on HuggingFace: * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8) * [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4) **Agentic workloads this model enables:** * **Computer Use Agent:** Perception loop for agents navigating GUIs — reads screens, understands UI state over time, validates outcomes. Collapses vision and reasoning into a single loop. * **Document Intelligence:** Interprets documents, charts, tables, screenshots, and mixed media inputs for enterprise analysis and compliance workflows. * **Audio & Video Understanding Agents:** Maintains continuous audio-video context for customer service, research, and monitoring workflows, tying what was said, shown, and documented into a single reasoning stream. ## 2. SGLang Installation Install SGLang via pip or from source: ```shell Command theme={null} # Install via pip pip install sglang # Or install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend:** **H100/H200:** Use flash attention 3 backend by default. **B200:** Use flashinfer backend by default. * **TP support:** To set tensor parallelism, use `--tp <1|2|4|8>`. A 4×H100 setup is recommended for the BF16 variant. * **FP8 KV cache:** To enable FP8 KV cache, append `--kv-cache-dtype fp8_e4m3`. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload. * **Reasoning parser:** Append `--reasoning-parser deepseek-r1` to enable structured reasoning traces (`reasoning_content` field in the response). * **Tool calling:** Append `--tool-call-parser qwen3_coder` to enable tool calling support. ## 4. Model Invocation The command below launches the server for a 4×H100 setup with reasoning and tool calling enabled. See [Section 4.8](#4-8-fp8-and-nvfp4-deployment) for FP8 and NVFP4 variants. ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` ### 4.1 Basic Usage (Text) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.reasoning_content, resp.choices[0].message.content) ``` Output: ```text Output theme={null} Reasoning: SGLang is a serving framework I know from my training data. Let me recall the key features... Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn and few-shot workloads. - **OpenAI-compatible API** — Drop-in replacement for the OpenAI Python client; no application code changes required to serve a locally-hosted model. - **High-throughput serving** — Continuous batching, chunked prefill, and optimized CUDA kernels deliver state-of-the-art throughput on NVIDIA GPUs across A100, H100, and B200. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"}, ], temperature=0.6, max_tokens=512, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` ### 4.2 Image Understanding Pass image inputs using the OpenAI vision format. Supports both URLs and base64-encoded images: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # From URL resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}, }, {"type": "text", "text": "Describe this image in detail."}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` For local images, encode as base64: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("screenshot.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}, }, {"type": "text", "text": "What UI elements are visible on this screen? What action would you take next?"}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.content) ``` ### 4.3 Video Understanding Nemotron 3 Nano Omni uses Conv3D layers and Efficient Video Sampling (EVS) for temporal-spatial video reasoning, processing longer videos at the same compute budget: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("video.mp4", "rb") as f: video_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": f"data:video/mp4;base64,{video_b64}"}, }, {"type": "text", "text": "Summarize what happens in this video step by step."}, ], } ], temperature=0.6, max_tokens=1024, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` ### 4.4 Audio Understanding Pass audio inputs as base64-encoded WAV or MP3 data: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}, }, {"type": "text", "text": "Transcribe and summarize what was said in this audio."}, ], } ], temperature=0.6, max_tokens=512, ) print(resp.choices[0].message.content) ``` ### 4.5 Mixed Multimodal Input Combine modalities in a single request. For example, an image alongside an audio question about it: ```python Example theme={null} import base64 from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("chart.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}, }, {"type": "text", "text": "Analyze this chart. What are the key trends and what conclusion does the data support?"}, ], } ], temperature=0.6, max_tokens=1024, ) print(resp.choices[0].message.reasoning_content) print(resp.choices[0].message.content) ``` ### 4.6 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. Toggle per-request by setting `enable_thinking` to `False`: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Reasoning ON (default) print("Reasoning on") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the derivative of x^3 sin(x)?"}, ], temperature=0.6, max_tokens=1024, ) print(f"Reasoning:\n{resp.choices[0].message.reasoning_content[:300]}...\nContent:\n{resp.choices[0].message.content}") print("\n") # Reasoning OFF print("Reasoning off") resp = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is 15% of 200?"}, ], temperature=0.6, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(f"Content:\n{resp.choices[0].message.content}") ``` Output: ```text Output theme={null} Reasoning on Reasoning: The user wants the derivative of x^3 sin(x). I'll apply the product rule: d/dx[u·v] = u'v + uv'. Here u = x^3, v = sin(x). So u' = 3x^2, v' = cos(x). The result is 3x^2·sin(x) + x^3·cos(x)... Content: Using the product rule: d/dx[x³ sin(x)] = 3x² sin(x) + x³ cos(x) Reasoning off Content: 15% of 200 is **30**. ``` ### 4.7 Tool Calling Call functions using the OpenAI Tools schema. The server must be launched with `--tool-call-parser qwen3_coder`: ```python Example theme={null} from openai import OpenAI SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16" client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") TOOLS = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and state, e.g. San Francisco, CA", }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], }, }, "required": ["location"], }, }, } ] completion = client.chat.completions.create( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the weather like in Santa Clara, CA?"}, ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False, ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user is asking about weather in Santa Clara, CA. I have a get_weather function that takes a location and optional unit. I should call it with location="Santa Clara, CA". [ChatCompletionMessageFunctionToolCall(id='call_abc123', function=Function(arguments='{"location": "Santa Clara, CA", "unit": "fahrenheit"}', name='get_weather'), type='function', index=0)] ``` ### 4.8 FP8 and NVFP4 Deployment **FP8 variant** (recommended for throughput-critical serving on H100/H200/B200): ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` **NVFP4 variant** (maximum efficiency on Blackwell B200): ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \ --host 0.0.0.0 \ --port 30000 \ --tp 4 \ --trust-remote-code \ --tool-call-parser qwen3_coder \ --reasoning-parser deepseek-r1 ``` *** ## 5. Benchmark ### 5.1 Efficiency Benchmark Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni models at the same interactivity level, delivering lower cost and better scalability without sacrificing responsiveness. It also achieves **\~20% higher multimodal intelligence** compared to the best open alternative across image, video, and audio reasoning tasks. ### 5.2 Speed Benchmark **Test Environment:** * Hardware: B200 (8×) * Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning * Tensor Parallelism: 4 * SGLang Version: main branch Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --trust-remote-code \ --tp 4 \ --max-running-requests 1024 \ --host 0.0.0.0 \ --attention-backend flashinfer \ --port 30000 ``` Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 206.52 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 1945477 Request throughput (req/s): 19.83 Input token throughput (tok/s): 10080.25 Output token throughput (tok/s): 10107.18 Peak output token throughput (tok/s): 20199.00 Peak concurrent requests: 291 Total token throughput (tok/s): 20187.44 Concurrency: 250.83 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12646.47 Median E2E Latency (ms): 12371.84 P90 E2E Latency (ms): 22889.81 P99 E2E Latency (ms): 26528.70 ---------------Time to First Token---------------- Mean TTFT (ms): 220.66 Median TTFT (ms): 97.67 P99 TTFT (ms): 2068.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 24.98 Median TPOT (ms): 24.36 P99 TPOT (ms): 44.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.43 Median ITL (ms): 10.91 P95 ITL (ms): 62.68 P99 ITL (ms): 100.60 Max ITL (ms): 2171.93 ================================================== ``` ### 5.3 Accuracy Benchmark **Environment** * Hardware: B200 (8×) * Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```shell Command theme={null} sglang serve \ --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \ --trust-remote-code \ --tp 4 \ --attention-backend flashinfer \ --reasoning-parser deepseek-r1 ``` #### 5.3.1 GSM8K Benchmark **Run Benchmark** ```shell Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.830 Invalid: 0.000 Latency: 13.970 s Output throughput: 1611.623 token/s ``` #### 5.3.2 MMLU Benchmark **Run Benchmark** ```shell Command theme={null} python3 benchmark/mmlu/bench_sglang.py --port 30000 ``` **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.510 subject: anatomy, #q:135, acc: 0.711 subject: astronomy, #q:152, acc: 0.829 subject: business_ethics, #q:100, acc: 0.760 subject: clinical_knowledge, #q:265, acc: 0.781 subject: college_biology, #q:144, acc: 0.854 subject: college_chemistry, #q:100, acc: 0.560 subject: college_computer_science, #q:100, acc: 0.700 subject: college_mathematics, #q:100, acc: 0.590 subject: college_medicine, #q:173, acc: 0.775 subject: college_physics, #q:102, acc: 0.559 subject: computer_security, #q:100, acc: 0.750 subject: conceptual_physics, #q:235, acc: 0.821 subject: econometrics, #q:114, acc: 0.605 subject: electrical_engineering, #q:145, acc: 0.759 subject: elementary_mathematics, #q:378, acc: 0.638 subject: formal_logic, #q:126, acc: 0.524 subject: global_facts, #q:100, acc: 0.400 subject: high_school_biology, #q:310, acc: 0.906 subject: high_school_chemistry, #q:203, acc: 0.759 subject: high_school_computer_science, #q:100, acc: 0.860 subject: high_school_european_history, #q:165, acc: 0.812 subject: high_school_geography, #q:198, acc: 0.889 subject: high_school_government_and_politics, #q:193, acc: 0.933 subject: high_school_macroeconomics, #q:390, acc: 0.785 subject: high_school_mathematics, #q:270, acc: 0.496 subject: high_school_microeconomics, #q:238, acc: 0.887 subject: high_school_physics, #q:151, acc: 0.675 subject: high_school_psychology, #q:545, acc: 0.895 subject: high_school_statistics, #q:216, acc: 0.731 subject: high_school_us_history, #q:204, acc: 0.858 subject: high_school_world_history, #q:237, acc: 0.873 subject: human_aging, #q:223, acc: 0.740 subject: human_sexuality, #q:131, acc: 0.855 subject: international_law, #q:121, acc: 0.851 subject: jurisprudence, #q:108, acc: 0.815 subject: logical_fallacies, #q:163, acc: 0.847 subject: machine_learning, #q:112, acc: 0.598 subject: management, #q:103, acc: 0.864 subject: marketing, #q:234, acc: 0.910 subject: medical_genetics, #q:100, acc: 0.880 subject: miscellaneous, #q:783, acc: 0.881 subject: moral_disputes, #q:346, acc: 0.780 subject: moral_scenarios, #q:895, acc: 0.543 subject: nutrition, #q:306, acc: 0.814 subject: philosophy, #q:311, acc: 0.733 subject: prehistory, #q:324, acc: 0.852 subject: professional_accounting, #q:282, acc: 0.553 subject: professional_law, #q:1534, acc: 0.565 subject: professional_medicine, #q:272, acc: 0.779 subject: professional_psychology, #q:612, acc: 0.760 subject: public_relations, #q:110, acc: 0.709 subject: security_studies, #q:245, acc: 0.759 subject: sociology, #q:201, acc: 0.831 subject: us_foreign_policy, #q:100, acc: 0.910 subject: virology, #q:166, acc: 0.560 subject: world_religions, #q:171, acc: 0.807 Total latency: 67.512 Average accuracy: 0.737 ``` # NVIDIA Nemotron3-Super Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Super ## 1. Model Introduction `NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step. Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack. * **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5). * **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text. * **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation. ## 2. SGLang Installation SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker. ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command. ### 3.2 Configuration Tips * **Attention backend**: **H200**: Use flash attention 3 backend by default. **B200**: Use flashinfer backend by default. * **TP support**: To set tp size, use `--tp <2|4|8>`. * **FP8 KV cache**: To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`. ## 4. Model Invocation ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --host 0.0.0.0 \ --port 5000 \ --trust-remote-code \ --tp 4 \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_3 ``` ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=1024, ) print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content) print("\n") ``` Output: ```text Output theme={null} Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others. First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens) Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens. - It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput. - Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` Output: ```text Output theme={null} The first 5 prime numbers are: **2, 3, 5, 7, 11**. ### Explanation: - A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself. - **2** is the smallest and only even prime number. - **3** is prime (divisible only by 1 and 3). - **4** is not prime (divisible by 2). - **5** is prime. - **6** is not prime (divisible by 2 and 3). - **7** is prime. - **8, 9, 10** are not prime. - **11** is prime (the fifth in the sequence). Note: **1 is not considered a prime number** by definition, as it has only one positive divisor. This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊 ``` ### 4.2 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."} ], temperature=1, max_tokens=1024, ) print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...") print("\n") # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Give me 3 facts about SGLang."} ], temperature=0, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...") ``` Output: ```text Output theme={null} Reasoning on Reasoning: We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs. Let's deci... Content: Silicon hearts beat Paint vivid worlds with bright light GPU dreams rise... Reasoning off Content: Certainly! Here are three accurate and informative facts about **SGLang**: 1. **SGLang is a high-performance serving system for large language models (LLMs)** Developed by researchers at UC Berk... ``` ### 4.3 Tool Calling Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:5000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": { "type": "integer", "description": "The total amount of the bill" }, "tip_percentage": { "type": "integer", "description": "The percentage of tip to be applied" } }, "required": ["bill_total", "tip_percentage"] } } } ] completion = client.chat.completions.create( model="nemotron", messages=[ {"role": "system", "content": ""}, {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that. [ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)] ``` ### 4.4 Controlling Reasoning Budget The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character. If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens. ```python Example theme={null} from typing import Any, Dict, List import openai from transformers import AutoTokenizer class ThinkingBudgetClient: def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str): self.base_url = base_url self.api_key = api_key self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path) self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key) def chat_completion( self, model: str, messages: List[Dict[str, Any]], reasoning_budget: int = 512, max_tokens: int = 1024, **kwargs, ) -> Dict[str, Any]: assert ( max_tokens > reasoning_budget ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}" # 1. first call chat completion to get reasoning content response = self.client.chat.completions.create( model=model, messages=messages, max_tokens=reasoning_budget, **kwargs ) reasoning_content = response.choices[0].message.reasoning_content or "" if "" not in reasoning_content: # reasoning content is too long, closed with a period (.) reasoning_content = f"{reasoning_content}.\n\n\n" reasoning_tokens_used = len( self.tokenizer.encode(reasoning_content, add_special_tokens=False) ) remaining_tokens = max_tokens - reasoning_tokens_used assert ( remaining_tokens > 0 ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget." # 2. append reasoning content to messages and call completion messages.append({"role": "assistant", "content": reasoning_content}) prompt = self.tokenizer.apply_chat_template( messages, tokenize=False, continue_final_message=True, ) response = self.client.completions.create( model=model, prompt=prompt, max_tokens=remaining_tokens, **kwargs ) response_data = { "reasoning_content": reasoning_content.strip().strip("").strip(), "content": response.choices[0].text, "finish_reason": response.choices[0].finish_reason, } return response_data ``` Usage example with `reasoning_budget=128`: ```python Example theme={null} SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" # Client client = ThinkingBudgetClient( base_url="http://127.0.0.1:5000/v1", api_key="null", tokenizer_name_or_path=SERVED_MODEL_NAME ) resp = client.chat_completion( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Write a haiku about GPUs."} ], temperature=1, max_tokens=512, reasoning_budget=128 ) print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"]) ``` Output: ```text Output theme={null} Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact. First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once. Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor. Content: Silicon minds awaken, Thousands of cores hum in unison— Lightning paints the void. ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: H200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 * Tensor Parallelism: 4 * SGLang Version: main branch * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --tp 4 \ --max-running-requests 1024 \ --host 0.0.0.0 \ --port 5000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 5000 \ --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 623.49 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 2044666 Request throughput (req/s): 6.57 Input token throughput (tok/s): 3338.85 Output token throughput (tok/s): 3347.77 Peak output token throughput (tok/s): 6349.00 Peak concurrent requests: 270 Total token throughput (tok/s): 6686.62 Concurrency: 250.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 38108.46 Median E2E Latency (ms): 37186.80 P90 E2E Latency (ms): 69325.24 P99 E2E Latency (ms): 77776.90 ---------------Time to First Token---------------- Mean TTFT (ms): 436.49 Median TTFT (ms): 114.90 P99 TTFT (ms): 6938.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 75.02 Median TPOT (ms): 76.02 P99 TPOT (ms): 92.27 ---------------Inter-Token Latency---------------- Mean ITL (ms): 74.07 Median ITL (ms): 38.45 P95 ITL (ms): 230.42 P99 ITL (ms): 242.70 Max ITL (ms): 7181.72 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: H200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \ --trust-remote-code \ --tp 4 \ --reasoning-parser nemotron_3 ``` **Run Benchmark** ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 5000 ``` **Test Results:** ```text Output theme={null} Accuracy: 0.950 Invalid: 0.000 Latency: 21.442 s Output throughput: 996.815 token/s ``` #### 5.2.2 MMLU Benchmark **Run Benchmark** ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py --port 5000 ``` **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.730 subject: anatomy, #q:135, acc: 0.830 subject: astronomy, #q:152, acc: 0.934 subject: business_ethics, #q:100, acc: 0.830 subject: clinical_knowledge, #q:265, acc: 0.879 subject: college_biology, #q:144, acc: 0.931 subject: college_chemistry, #q:100, acc: 0.620 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.820 subject: college_medicine, #q:173, acc: 0.821 subject: college_physics, #q:102, acc: 0.794 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.919 subject: econometrics, #q:114, acc: 0.746 subject: electrical_engineering, #q:145, acc: 0.828 subject: elementary_mathematics, #q:378, acc: 0.926 subject: formal_logic, #q:126, acc: 0.857 subject: global_facts, #q:100, acc: 0.570 subject: high_school_biology, #q:310, acc: 0.952 subject: high_school_chemistry, #q:203, acc: 0.828 subject: high_school_computer_science, #q:100, acc: 0.940 subject: high_school_european_history, #q:165, acc: 0.861 subject: high_school_geography, #q:198, acc: 0.939 subject: high_school_government_and_politics, #q:193, acc: 0.990 subject: high_school_macroeconomics, #q:390, acc: 0.928 subject: high_school_mathematics, #q:270, acc: 0.700 subject: high_school_microeconomics, #q:238, acc: 0.966 subject: high_school_physics, #q:151, acc: 0.834 subject: high_school_psychology, #q:545, acc: 0.960 subject: high_school_statistics, #q:216, acc: 0.852 subject: high_school_us_history, #q:204, acc: 0.926 subject: high_school_world_history, #q:237, acc: 0.937 subject: human_aging, #q:223, acc: 0.879 subject: human_sexuality, #q:131, acc: 0.939 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.898 subject: logical_fallacies, #q:163, acc: 0.914 subject: machine_learning, #q:112, acc: 0.821 subject: management, #q:103, acc: 0.903 subject: marketing, #q:234, acc: 0.944 subject: medical_genetics, #q:100, acc: 0.980 subject: miscellaneous, #q:783, acc: 0.945 subject: moral_disputes, #q:346, acc: 0.861 subject: moral_scenarios, #q:895, acc: 0.542 subject: nutrition, #q:306, acc: 0.902 subject: philosophy, #q:311, acc: 0.884 subject: prehistory, #q:324, acc: 0.920 subject: professional_accounting, #q:282, acc: 0.805 subject: professional_law, #q:1534, acc: 0.681 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.889 subject: public_relations, #q:110, acc: 0.800 subject: security_studies, #q:245, acc: 0.837 subject: sociology, #q:201, acc: 0.960 subject: us_foreign_policy, #q:100, acc: 0.920 subject: virology, #q:166, acc: 0.590 subject: world_religions, #q:171, acc: 0.906 Total latency: 150.267 Average accuracy: 0.841 ``` # NVIDIA Nemotron3-Ultra Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra Deploy NVIDIA Nemotron3-Ultra with SGLang - 550B hybrid MoE model (55B active) with 1M context window, BF16/NVFP4 support, built for long-running autonomous agents. ## 1. Model Introduction `NVIDIA Nemotron3-Ultra` is an open frontier reasoning model in the Nemotron 3 family, built for long-running autonomous agents. It is optimized for complex orchestration across coding, deep research, enterprise workflows, and EDA use cases where agents must sustain reasoning across many steps and large context windows. Nemotron 3 Ultra is a 550B parameter hybrid MoE model that activates only 55B parameters per forward pass, delivering frontier reasoning accuracy with high-throughput inference. It supports a 1M token context window so agents can keep conversation history, tool outputs, and plan state in view across persistent workflows. Architecture and key features: * **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack. * **Long-horizon agentic reasoning:** Tuned for agents that plan, call tools, inspect results, recover from failures, and continue working across long task horizons — coding, deep research, enterprise automation, and EDA. * **1M token context window:** Sustains coherent agent state across extended workflows without re-ingestion. * **BF16 and NVFP4 quantization:** Deployable from multi-node H100 down to a single Blackwell node with NVFP4. * **Multi-environment RL post-training:** Post-trained with reinforcement learning across multiple environments for robust reasoning and reliable agentic behavior. * **Open weights, open data, open recipes:** Customizable for domain-specific agents and deployable across your own infrastructure. **Modalities:** Input: text — Output: text **Supported GPUs:** * **BF16:** 16×H100, 16×H200, 8×B200/B300 * **NVFP4:** 4/8×B200/B300, 4×GB200/GB300 Available model variants on HuggingFace: * [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16) * [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4) ## 2. SGLang Installation Nemotron3-Ultra support is included in the latest stable release. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance tuning. ### 3.1 Basic Configuration **Interactive Command Generator**: select model precision, hardware, tensor parallelism, and common knobs to generate a launch command. The generator only emits a runnable command for combinations that NVIDIA / SGLang have validated. Selecting an unverified tuple (e.g. NVFP4 on H100/H200, BF16 with TP=4 on H100, …) is **blocked** — the command pane shows an explicit error and the verified support matrix instead of a launch line, so unvalidated commands can't be copied by accident. ### 3.2 Configuration Tips * **Attention backend**: **H100/H200**: Use flash attention 3 backend by default. **B200/GB200/B300/GB300**: Set `--attention-backend trtllm_mha`. The flashinfer default breaks the overlap scheduler on Blackwell, so `trtllm_mha` is required there. * **Mamba scheduler strategy**: Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state. * **Mamba backend**: The Mamba layers use the Triton SSM kernels by default. For better performance, set `--mamba-backend flashinfer` to use the FlashInfer Mamba kernels instead. * **Mamba SSM precision**: The SSM state dtype defaults to the model config value. Set `--mamba-ssm-dtype float16` to store the Mamba states in FP16, which reduces mamba cache memory without significant accuracy loss. * **Mamba SSM stochastic rounding**: When storing the Mamba states in FP16, add `--enable-mamba-cache-stochastic-rounding` to round SSM cache writes stochastically and reduce accumulation bias. It requires `--mamba-ssm-dtype float16` and CUDA; with the default `--mamba-backend triton` it additionally requires SM100. Use `--mamba-cache-philox-rounds` to control the number of Philox rounds (`0` uses the backend default). * **TP support**: To set tp size, use `--tp <4|8|16>`. Recommended pairings: * BF16: `--tp 16` on H100/H200, `--tp 8` on B200/B300 * NVFP4: `--tp 4` or `--tp 8` on B200/B300, `--tp 4` on GB200/GB300 * **Multi-node BF16 on H100**: The 16×H100 BF16 setup spans two nodes. Use `--dist-init-addr :5000 --nnodes 2 --node-rank <0|1>` on each node and keep `--tp 16`. * **DP attention**: By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--dp --enable-dp-attention`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--dp` must divide `--tp`. * **Expert parallel (EP)**: This MoE only supports `ep_size == 1` (off) or `ep_size == tp_size`. To enable expert parallelism, set `--ep ` with the same value as `--tp`. * **Multi-token prediction (MTP)**: Enable MTP speculative decoding (the toggle above) for low latency. * **FP8 KV cache**: To enable fp8 kv cache, set `--kv-cache-dtype fp8_e4m3`. This is enabled by default on the NVFP4 checkpoint. * **Reasoning parser**: Set `--reasoning-parser nemotron_3` to enable structured reasoning traces (`reasoning_content` field in the response). * **Tool calling**: Set `--tool-call-parser qwen3_coder` to enable tool calling support. ## 4. Model Invocation ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \ --trust-remote-code \ --tp 8 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_3 ``` ### 4.1 Basic Usage (OpenAI-Compatible API) SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Give me 3 bullet points about SGLang."}, ], temperature=0.6, max_tokens=1024, ) print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content) print("\n") ``` Output: ```text Output theme={null} Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens) Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads. - **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling. - **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200. ``` Streaming chat completion: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) stream = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "What are the first 5 prime numbers?"} ], temperature=0.7, max_tokens=1024, stream=True, ) for chunk in stream: delta = chunk.choices[0].delta if delta and delta.content: print(delta.content, end="", flush=True) ``` Output: ```text Output theme={null} The first 5 prime numbers are: **2, 3, 5, 7, 11**. ### Explanation: - A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself. - **2** is the smallest prime and the only even prime. - **3, 5, 7, 11** are each divisible only by 1 and themselves. - **1** is not prime by definition (it has only one positive divisor). - **4, 6, 8, 9, 10** are composite. ``` ### 4.2 Reasoning The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Reasoning on (default) print("Reasoning on") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."} ], temperature=1, max_tokens=1024, ) print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...") print("\n") # Reasoning off print("Reasoning off") resp = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Give me 3 facts about SGLang."} ], temperature=0, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) print(f"Content: \n{resp.choices[0].message.content[:200]}...") ``` Output: ```text Output theme={null} Reasoning on Reasoning: The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t... Content: 1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering. 2. **Isolate state** — re-run with... Reasoning off Content: Here are 3 facts about SGLang: 1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale. ... ``` ### 4.3 Tool Calling Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. The server must be launched with `--tool-call-parser qwen3_coder`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) # Tool calling via OpenAI tools schema TOOLS = [ { "type": "function", "function": { "name": "search_codebase", "description": "Search the project codebase for a symbol or pattern.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "The symbol, function name, or regex to search for" }, "path": { "type": "string", "description": "Optional sub-path to restrict the search to" } }, "required": ["query"] } } } ] completion = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", messages=[ {"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."}, {"role": "user", "content": "Where is the `RadixCache` class defined?"} ], tools=TOOLS, temperature=0.6, top_p=0.95, max_tokens=512, stream=False ) print(completion.choices[0].message.reasoning_content) print(completion.choices[0].message.tool_calls) ``` Output: ```text Output theme={null} The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query. [ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)] ``` ### 4.4 Controlling Reasoning Budget The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character. If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens. ```python Example theme={null} from typing import Any, Dict, List import openai from transformers import AutoTokenizer class ThinkingBudgetClient: def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str): self.base_url = base_url self.api_key = api_key self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path) self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key) def chat_completion( self, model: str, messages: List[Dict[str, Any]], reasoning_budget: int = 512, max_tokens: int = 1024, **kwargs, ) -> Dict[str, Any]: assert ( max_tokens > reasoning_budget ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}" # 1. first call chat completion to get reasoning content response = self.client.chat.completions.create( model=model, messages=messages, max_tokens=reasoning_budget, **kwargs ) reasoning_content = response.choices[0].message.reasoning_content or "" if "" not in reasoning_content: # reasoning content is too long, closed with a period (.) reasoning_content = f"{reasoning_content}.\n\n\n" reasoning_tokens_used = len( self.tokenizer.encode(reasoning_content, add_special_tokens=False) ) remaining_tokens = max_tokens - reasoning_tokens_used assert ( remaining_tokens > 0 ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget." # 2. append reasoning content to messages and call completion messages.append({"role": "assistant", "content": reasoning_content}) prompt = self.tokenizer.apply_chat_template( messages, tokenize=False, continue_final_message=True, ) response = self.client.completions.create( model=model, prompt=prompt, max_tokens=remaining_tokens, **kwargs ) response_data = { "reasoning_content": reasoning_content.strip().strip("").strip(), "content": response.choices[0].text, "finish_reason": response.choices[0].finish_reason, } return response_data ``` Usage example with `reasoning_budget=256`: ```python Example theme={null} SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16" # Client client = ThinkingBudgetClient( base_url="http://127.0.0.1:30000/v1", api_key="null", tokenizer_name_or_path=SERVED_MODEL_NAME ) resp = client.chat_completion( model=SERVED_MODEL_NAME, messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."} ], temperature=1, max_tokens=1024, reasoning_budget=256 ) print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"]) ``` Output: ```text Output theme={null} Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256... Content: **Research plan** 1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy. 2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant. 3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy. 4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets. 5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise. ``` *** ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: GB200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 * Tensor Parallelism: 4 * SGLang Version: main branch * Model Deployment Command: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --max-running-requests 1024 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 4096 \ --max-concurrency 256 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 256 Successful requests: 4096 Benchmark duration (s): 1184.58 Total input tokens: 2081726 Total input text tokens: 2081726 Total generated tokens: 2087288 Total generated tokens (retokenized): 1990224 Request throughput (req/s): 3.46 Input token throughput (tok/s): 1757.35 Output token throughput (tok/s): 1762.05 Peak output token throughput (tok/s): 3150.00 Peak concurrent requests: 266 Total token throughput (tok/s): 3519.40 Concurrency: 249.55 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 72169.95 Median E2E Latency (ms): 71994.47 P90 E2E Latency (ms): 99898.56 P99 E2E Latency (ms): 107119.61 ---------------Time to First Token---------------- Mean TTFT (ms): 40057.33 Median TTFT (ms): 41375.93 P99 TTFT (ms): 46377.89 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 63.15 Median TPOT (ms): 63.65 P99 TPOT (ms): 78.16 ---------------Inter-Token Latency---------------- Mean ITL (ms): 63.14 Median ITL (ms): 35.92 P95 ITL (ms): 178.10 P99 ITL (ms): 182.10 Max ITL (ms): 2466.36 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark **Environment** * Hardware: GB200 (4x) * Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 * Tensor Parallelism: 4 * SGLang Version: main branch **Launch Model** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --reasoning-parser nemotron_3 ``` **Run Benchmark** ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py ``` **Test Results:** ```text Output theme={null} Accuracy: 0.970 Invalid: 0.000 Latency: 29.129 s Output throughput: 745.333 token/s ``` #### 5.2.2 MMLU Benchmark **Run Benchmark** ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py ``` **Test Results:** ```text Output theme={null} TBD ``` *** # Nemotron3.5-Lightning Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3.5-Lightning Deploy NVIDIA Nemotron 3.5 Lightning with SGLang — NVFP4 serving with MTP, DFlash, and DSpark speculative decoding, reasoning, and tool calling. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv SGLANG_BUILD_RUST_EXTS=none uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git@refs/pull/33554/head#subdirectory=python' ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-nemotron3-5-lightning ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware and recipe to generate the launch command. Every platform publishes four operating points: **Balanced** (no speculation) plus three speculative decoders — **MTP**, **DFlash**, and **DSpark**. Use the Playground below to explore knobs beyond them. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **NVIDIA Nemotron 3.5 Lightning** is a 30B-A3B hybrid reasoning LLM. See the Hugging Face model cards below for architecture and evaluation details.
Checkpoint Precision Use
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 NVFP4 Serving — the checkpoint this page deploys
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 BF16 Full-precision reference
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash W4A16 DFlash speculative draft model
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark W4A16 DSpark speculative draft model
MTP needs no separate download — the draft head is embedded in the target checkpoint. ## 2. Usage The server speaks the OpenAI API. With `--reasoning-parser nemotron_3` enabled, the thinking trace lands in `message.reasoning_content` and the answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8000/v1", api_key="null", ) response = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Briefly explain: what is SGLang?"}, ], temperature=1.0, top_p=0.95, max_tokens=1024, ) choice = response.choices[0] print("Reasoning:", choice.message.reasoning_content) print("Content:", choice.message.content) ``` ### 2.1 Tool Calling With `--tool-call-parser qwen3_coder` enabled, structured tool calls are returned in `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://127.0.0.1:8000/v1", api_key="null", ) TOOLS = [ { "type": "function", "function": { "name": "calculate_tip", "parameters": { "type": "object", "properties": { "bill_total": {"type": "integer", "description": "The total amount of the bill"}, "tip_percentage": {"type": "integer", "description": "The percentage of tip to be applied"}, }, "required": ["bill_total", "tip_percentage"], }, }, } ] response = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", messages=[{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}], tools=TOOLS, max_tokens=1024, ) choice = response.choices[0] print("Content:", choice.message.content) print("Tool calls:", choice.message.tool_calls) ``` # GPT-OSS Source: https://docs.sglang.io/cookbook/autoregressive/OpenAI/GPT-OSS ## 1.Model Introduction [GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes. * **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters) * **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters) GPT-OSS introduces several groundbreaking innovations: * **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs. * **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users. * **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning. * **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs. * **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization. ## 2.SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3.Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Native web search:** Set `EXA_API_KEY` in the SGLang server environment to enable built-in web search (Exa). No `--tool-server` is required, and requests are tagged with `x-exa-integration: sglang`. * **Web search defaults:** `numResults=10`, search `type="auto"`, and `contents.highlights=true`. Override with `SGLANG_EXA_NUM_RESULTS`, `SGLANG_EXA_SEARCH_TYPE`, and `SGLANG_EXA_INCLUDE_HIGHLIGHTS`. * **Python tool:** Add `--tool-server demo` to enable the Python interpreter. Runs in a Docker sandbox by default; set `PYTHON_EXECUTION_BACKEND=UV` to run on the host (model-generated code executes locally — use with care). * **MCP tool servers:** For production, point SGLang at external MCP SSE servers with `--tool-server ip-1:port-1,ip-2:port-2`. * **Responses API:** GPT-OSS supports OpenAI's Responses API (`client.responses.create`) in addition to the standard Chat Completions API (see section 4.2.4). * **Use Python 3.12** when running the demo Python tool. * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4.Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --reasoning-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer. =============== Content ================= **Step‑by‑step solution** 1. **Understand what “percent” means** “15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\). 2. **Convert the percent to a decimal (or fraction)** \[ \frac{15}{100}=0.15 \] 3. **Set up the multiplication** To find 15 % of 240 we multiply 240 by the decimal 0.15: \[ 240 \times 0.15 \] 4. **Do the multiplication** One convenient way is to break it into two easier parts: \[ 240 \times 0.15 = 240 \times \left(\frac{15}{100}\right) = \frac{240 \times 15}{100} \] - First compute \(240 \times 15\): \[ 240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600 \] - Then divide by 100: \[ \frac{3600}{100} = 36 \] 5. **Write the result** \[ 15\% \text{ of } 240 = 36 \] --- **Answer:** \(36\) ``` #### 4.2.2 Tool Calling GPT-OSS supports tool calling capabilities. Enable the tool call parser: **Python Example (without Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tool-call-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Python Example (with Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --reasoning-parser gpt-oss \ --tool-call-parser gpt-oss \ --tp 8 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="openai/gpt-oss-120b", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!" ``` #### 4.2.3 EAGLE3 Speculative Decoding SGLang supports speculative decoding for GPT-OSS models using the EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes. ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \ --tp 2 ``` The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable. #### 4.2.4 Responses API and Built-in Tools GPT-OSS supports the OpenAI Responses API with built-in tool use (web search and Python interpreter). Set `EXA_API_KEY` to enable native web search; add `--tool-server demo` only when you also want the Python tool: ```shell Command theme={null} export EXA_API_KEY=YOUR_EXA_KEY # Optional: server-side Exa tuning (defaults shown) export SGLANG_EXA_NUM_RESULTS=10 export SGLANG_EXA_SEARCH_TYPE=auto export SGLANG_EXA_INCLUDE_HIGHLIGHTS=true # Optional: run Python tool on host instead of Docker (model code executes locally) export PYTHON_EXECUTION_BACKEND=UV python3 -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --tp 2 ``` For production, use external MCP SSE servers instead of `demo`: ```shell Command theme={null} mcp run -t sse browser_server.py:mcp mcp run -t sse python_server.py:mcp python -m sglang.launch_server \ --model-path openai/gpt-oss-120b \ --tool-server ip-1:port-1,ip-2:port-2 \ --tp 2 ``` **Example using Responses API:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="sk-123456") search_tools = [{"type": "web_search"}] python_tools = [{"type": "code_interpreter"}] # Configurable reasoning effort: "high", "medium", or "low" response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant.", reasoning_effort="high", input="In one sentence, explain the transformer architecture.", ) print(response.output_text) # Web search (requires EXA_API_KEY on the SGLang server) response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant, you can search the web when needed.", input="Search the web for the latest news about Nvidia stock price", tools=search_tools, ) print(response.output_text) # Python tool (requires launching SGLang with --tool-server demo) response = client.responses.create( model="openai/gpt-oss-120b", instructions="You are a helpful assistant, you could use python tool to execute code.", input="Use python tool to calculate the sum of 29138749187 and 29138749187", tools=python_tools, ) print(response.output_text) # Output: The sum is 58,277,498,374. ``` ## 5.Benchmark ### 5.1 Speed Benchmark * Hardware: NVIDIA B200 GPU (8x) * Tensor Parallelism: 8 * Model: openai/gpt-oss-120b * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 100 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 52.35 Total input tokens: 33178 Total input text tokens: 33178 Total input vision tokens: 0 Total generated tokens: 21251 Total generated tokens (retokenized): 20868 Request throughput (req/s): 1.91 Input token throughput (tok/s): 633.76 Output token throughput (tok/s): 405.93 Peak output token throughput (tok/s): 433.00 Peak concurrent requests: 8 Total token throughput (tok/s): 1039.69 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 523.30 Median E2E Latency (ms): 389.91 ---------------Time to First Token---------------- Mean TTFT (ms): 33.71 Median TTFT (ms): 31.79 P99 TTFT (ms): 108.98 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 2.31 Median TPOT (ms): 2.31 P99 TPOT (ms): 2.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 2.31 Median ITL (ms): 2.31 P95 ITL (ms): 2.35 P99 ITL (ms): 2.38 Max ITL (ms): 3.54 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model openai/gpt-oss-120b \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 24.76 Total input tokens: 297156 Total input text tokens: 297156 Total input vision tokens: 0 Total generated tokens: 192432 Total generated tokens (retokenized): 187145 Request throughput (req/s): 40.39 Input token throughput (tok/s): 12003.57 Output token throughput (tok/s): 7773.26 Peak output token throughput (tok/s): 13780.00 Peak concurrent requests: 156 Total token throughput (tok/s): 19776.83 Concurrency: 89.23 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2208.97 Median E2E Latency (ms): 1591.11 ---------------Time to First Token---------------- Mean TTFT (ms): 102.94 Median TTFT (ms): 31.53 P99 TTFT (ms): 674.32 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.31 Median TPOT (ms): 11.00 P99 TPOT (ms): 91.28 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.00 Median ITL (ms): 5.75 P95 ITL (ms): 25.35 P99 ITL (ms): 43.18 Max ITL (ms): 621.42 ================================================== ``` ### 5.2 Accuracy Benchmark ### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000 ``` * **Results**: * GPT-OSS-120b ```text Output theme={null} Accuracy: 0.880 Invalid: 0.005 Latency: 5.262 s Output throughput: 12143.675 token/s ``` * GPT-OSS-20b ```text Output theme={null} Accuracy: 0.535 Invalid: 0.165 Latency: 4.157 s Output throughput: 19589.165 token/s ``` # MiniCPM-V 4.6 Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6 ## 1. Model Introduction MiniCPM-V 4.6 is the next-generation multimodal model from [OpenBMB](https://huggingface.co/openbmb), the team behind the MiniCPM-V series. The model combines a **Qwen3.5-style hybrid LLM backbone** (Gated Delta Net + full attention) with a **NaViT-packed vision encoder** that handles arbitrary aspect ratios and high-resolution slicing natively, plus end-to-end video support. OpenBMB ships two variants on HuggingFace: * [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — base instruct model. Use this for general multimodal serving; thinking mode is still available per-request via `chat_template_kwargs.enable_thinking=true`. * [`openbmb/MiniCPM-V-4.6-Thinking`](https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking) — thinking-tuned variant with stronger chain-of-thought behavior. Pair with the same `--reasoning-parser qwen3` flag. **Key Features:** * **Hybrid LLM backbone**: Qwen3.5-style mix of Gated Delta Net (linear-attention) layers and full-attention layers, providing long-context efficiency without giving up modeling power. * **Native variable-resolution vision**: NaViT-packed vision encoder with mid-ViT merger and per-image window attention. Images of any aspect ratio are processed without forced letterboxing. * **High-resolution slicing**: Source image plus a configurable grid of slice tiles (up to 9 tiles in the open test variant) lets the model reason over fine detail in 1280×720+ images. * **Video**: Frame-by-frame multi-modal data items routed through the same vision encoder; any number of frames per request. * **Reasoning Parser**: switchable thinking mode (Qwen3.5 lineage), exposed via `chat_template_kwargs.enable_thinking` per request and SGLang's `--reasoning-parser qwen3` on the server side. * **Tool Calling**: Qwen3.5-style `…` XML format, surfaced as OpenAI-compatible `message.tool_calls` via SGLang's `--tool-call-parser qwen3_coder`. Composes with thinking mode and with image / video inputs. **License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). ## 2. SGLang Installation Pull the nightly Docker image (rolling tag, tracks `main`): ```bash theme={null} # CUDA 13 (Hopper / Blackwell, default) docker pull lmsysorg/sglang:dev # CUDA 12 (Ampere or older drivers) docker pull lmsysorg/sglang:dev-cu12 ``` For the general SGLang installation guide (PyPI, source, Docker) see the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate the appropriate deployment command. The `Variant` toggle switches between `openbmb/MiniCPM-V-4.6` (base) and `openbmb/MiniCPM-V-4.6-Thinking`. The `Reasoning Parser` and `Tool Call Parser` toggles add `--reasoning-parser qwen3` and `--tool-call-parser qwen3_coder` respectively; see §4.4 for usage details. ### 3.2 Configuration Tips * **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Multi-image and high-resolution images: the image processor produces one source patch plus per-slice tile patches; each is its own `MultimodalDataItem`. No special server-side flag needed. * Video: decoded frame-by-frame through the same image-style slicer. No extra flag needed; pass `video_url` in the OpenAI chat completion request. * **Chunked Prefill**: For high-concurrency vision benchmarking with many large/sliced images, pass `--chunked-prefill-size -1` to disable prefill chunking. The default chunked-prefill path can mis-split a request across an image boundary in `mm_utils.embed_mm_inputs` and crash the server; disabling chunking sidesteps this at the cost of higher TTFT under concurrency. For interactive serving leave the default on. ## 4. Model Invocation Deploy the model on an H200: ```bash Command theme={null} sglang serve --model-path openbmb/MiniCPM-V-4.6 \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.15 \ --mamba-radix-cache-strategy extra_buffer \ --page-size 64 \ --host 0.0.0.0 --port 30000 ``` ### 4.1 Basic Usage (Image) ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.ilankelman.org/stopsigns/australia.jpg", }, }, {"type": "text", "text": "Describe this image in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} A black SUV drives past a Chinese-style gate with a red stop sign and traditional architecture, while storefronts and street signs line the sidewalk. ``` ### 4.2 High-Resolution / Sliced Images The image processor automatically picks a slice grid (up to 9 tiles) for high-resolution inputs. A 1280×720 source produces grid `[2, 3]` * 7 patches with `tgt_sizes=[(24, 44), 6×(28, 36)]`, byte-for-byte matching the HF reference implementation. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg", }, }, {"type": "text", "text": "Describe this image in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} The Statue of Liberty stands tall against a cloudy sky, holding a torch aloft and a document in her left hand, symbolizing freedom and enlightenment. ``` ### 4.3 Video Input ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": ""}, }, {"type": "text", "text": "Describe what happens in this video in one sentence."}, ], } ], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) print(response.choices[0].message.content) ``` **Output Example** (run against an 8-frame synthetic test mp4 of shifting colored squares): ```text Output theme={null} The video shows a grid of colored squares moving in a random pattern. ``` ### 4.4 Advanced Usage #### 4.4.1 Reasoning Parser Pass `--reasoning-parser qwen3` to the server (toggle "Reasoning Parser" on in §3.1, default) so SGLang splits each response on the `` / `` boundaries: the pre-`` block goes to `reasoning_content`, the post-`` text to `content`. Per-request, the chat template's `enable_thinking` flag toggles whether the model actually emits reasoning. * **Thinking mode** (default, `enable_thinking=true`): assistant prompt ends with `\n`; the model writes reasoning, closes with ``, then the answer. `reasoning_content` and `content` are both populated. * **Instruct mode** (`enable_thinking=false`): the chat template injects an empty `` placeholder so the model emits no thinking tokens; `reasoning_content` ends up empty. ```python Example (thinking mode) theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}], max_tokens=200, ) msg = response.choices[0].message print("reasoning_content:", msg.reasoning_content) print("content :", msg.content) ``` ```text Output theme={null} reasoning_content: Got it, let's see. The user wants a reply with "hi" and no explanation. So I need to just say "hi" as the response. ... content : hi ``` ```python Example (instruct mode) theme={null} response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}], max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) msg = response.choices[0].message print("reasoning_content:", msg.reasoning_content) print("content :", msg.content) ``` ```text Output theme={null} reasoning_content: content : hi ``` #### 4.4.2 Tool Calling Pass `--tool-call-parser qwen3_coder` to the server (toggle "Tool Call Parser" on in §3.1) so SGLang extracts `` blocks from the model output into the OpenAI-style `message.tool_calls` field (with `finish_reason="tool_calls"`). The model speaks the Qwen3.5 XML tool-call format (`v`); the `qwen3_coder` parser is the right one. Tool calls compose with both reasoning modes and with image / video inputs. Do **not** use `--tool-call-parser qwen` for MiniCPM-V 4.6 — that parser expects the older Qwen2.5 JSON format `{"name":..., "arguments":...}`, but both public 4.6 variants emit the Qwen3.5-style XML format with nested `` and `` tags. With `qwen` the outer `` markers match but the inner JSON parse fails, so `tool_calls` returns empty and the raw markup is left in `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "location": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, }, ] response = client.chat.completions.create( model="openbmb/MiniCPM-V-4.6", messages=[{"role": "user", "content": "What is the weather in San Francisco? Use the tool."}], tools=tools, max_tokens=200, extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) choice = response.choices[0] print("finish_reason:", choice.finish_reason) for tc in choice.message.tool_calls or []: print(f" {tc.function.name}({tc.function.arguments})") ``` ```text Output theme={null} finish_reason: tool_calls get_weather({"location": "San Francisco", "unit": "celsius"}) ``` To get the final natural-language answer, feed the tool's result back as a `tool` role message and call the API again with the same `tools` list — the model emits `finish_reason="stop"` with the answer in `content`. ## 5. Benchmark **Common Test Environment (all benchmarks below):** * Hardware: 1× NVIDIA H200 (141 GB), single GPU (no TP / DP) * Docker Image: `lmsysorg/sglang:dev` (transformers 5.6.0, sgl-kernel 0.4.2.post1) * Precision: BF16 **Common Server Launch Command:** ```bash Command theme={null} CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \ --model-path openbmb/MiniCPM-V-4.6 \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.5 \ --mamba-radix-cache-strategy extra_buffer \ --chunked-prefill-size -1 \ --host 0.0.0.0 --port 30000 ``` (`--chunked-prefill-size -1` is required for the vision throughput run; see §3.2.) ### 5.1 Accuracy Benchmark #### 5.1.1 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 48 --max-new-tokens 2048 ``` * Test Result ``` {'Accounting': {'acc': 0.767, 'num': 30}, 'Agriculture': {'acc': 0.533, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30}, 'Art': {'acc': 0.6, 'num': 30}, 'Art_Theory': {'acc': 0.667, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.533, 'num': 30}, 'Biology': {'acc': 0.333, 'num': 30}, 'Chemistry': {'acc': 0.333, 'num': 30}, 'Clinical_Medicine': {'acc': 0.467, 'num': 30}, 'Computer_Science': {'acc': 0.333, 'num': 30}, 'Design': {'acc': 0.533, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.333, 'num': 30}, 'Economics': {'acc': 0.633, 'num': 30}, 'Electronics': {'acc': 0.5, 'num': 30}, 'Energy_and_Power': {'acc': 0.633, 'num': 30}, 'Finance': {'acc': 0.533, 'num': 30}, 'Geography': {'acc': 0.367, 'num': 30}, 'History': {'acc': 0.533, 'num': 30}, 'Literature': {'acc': 0.7, 'num': 30}, 'Manage': {'acc': 0.367, 'num': 30}, 'Marketing': {'acc': 0.733, 'num': 30}, 'Materials': {'acc': 0.367, 'num': 30}, 'Math': {'acc': 0.567, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.333, 'num': 30}, 'Music': {'acc': 0.267, 'num': 30}, 'Overall': {'acc': 0.527, 'num': 900}, 'Overall-Art and Design': {'acc': 0.517, 'num': 120}, 'Overall-Business': {'acc': 0.607, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.553, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.617, 'num': 120}, 'Overall-Science': {'acc': 0.473, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210}, 'Pharmacy': {'acc': 0.667, 'num': 30}, 'Physics': {'acc': 0.767, 'num': 30}, 'Psychology': {'acc': 0.567, 'num': 30}, 'Public_Health': {'acc': 0.767, 'num': 30}, 'Sociology': {'acc': 0.667, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.527 ``` ### 5.2 Speed Benchmark We use SGLang's built-in `bench_serving` tool with random text prompts (1000 input / 1000 output tokens) to characterize text-only serving performance. #### 5.2.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 7.47 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 3554 Request throughput (req/s): 1.34 Input token throughput (tok/s): 816.44 Output token throughput (tok/s): 564.73 Peak output token throughput (tok/s): 690.00 Peak concurrent requests: 4 Total token throughput (tok/s): 1381.17 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 746.20 Median E2E Latency (ms): 590.05 P90 E2E Latency (ms): 1446.13 P99 E2E Latency (ms): 1709.38 ---------------Time to First Token---------------- Mean TTFT (ms): 138.12 Median TTFT (ms): 103.70 P99 TTFT (ms): 330.79 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.44 Median TPOT (ms): 1.44 P99 TPOT (ms): 1.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.44 Median ITL (ms): 1.45 P95 ITL (ms): 1.49 P99 ITL (ms): 1.57 Max ITL (ms): 5.79 ================================================== ``` #### 5.2.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 47.07 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 469844 Request throughput (req/s): 21.24 Input token throughput (tok/s): 10675.32 Output token throughput (tok/s): 10627.69 Peak output token throughput (tok/s): 25911.00 Peak concurrent requests: 130 Total token throughput (tok/s): 21303.01 Concurrency: 97.24 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4576.94 Median E2E Latency (ms): 4331.97 P90 E2E Latency (ms): 8634.07 P99 E2E Latency (ms): 9636.44 ---------------Time to First Token---------------- Mean TTFT (ms): 206.50 Median TTFT (ms): 184.72 P99 TTFT (ms): 624.23 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.73 Median TPOT (ms): 9.16 P99 TPOT (ms): 13.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.75 Median ITL (ms): 0.05 P95 ITL (ms): 29.95 P99 ITL (ms): 108.91 Max ITL (ms): 448.40 ================================================== ``` ### 5.3 Vision Speed Benchmark We use SGLang's built-in `bench_serving` tool with random images. Each request has 128 input text tokens, one 720p image, and 1024 output tokens. #### 5.3.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name image \ --image-count 1 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 10.26 Total input tokens: 767 Total input text tokens: 750 Total input vision tokens: 17 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.97 Input token throughput (tok/s): 74.77 Output token throughput (tok/s): 411.39 Peak output token throughput (tok/s): 654.00 Peak concurrent requests: 2 Total token throughput (tok/s): 486.16 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1024.04 Median E2E Latency (ms): 897.99 P90 E2E Latency (ms): 1584.25 P99 E2E Latency (ms): 1781.78 ---------------Time to First Token---------------- Mean TTFT (ms): 416.94 Median TTFT (ms): 403.18 P99 TTFT (ms): 477.49 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.44 Median TPOT (ms): 1.44 P99 TPOT (ms): 1.45 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.44 Median ITL (ms): 1.44 P95 ITL (ms): 1.48 P99 ITL (ms): 1.56 Max ITL (ms): 2.89 ================================================== ``` #### 5.3.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model openbmb/MiniCPM-V-4.6 \ --dataset-name image \ --image-count 1 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 360.01 Total input tokens: 79925 Total input text tokens: 78283 Total input vision tokens: 1642 Total generated tokens: 510855 Total generated tokens (retokenized): 430289 Request throughput (req/s): 2.78 Input token throughput (tok/s): 222.01 Output token throughput (tok/s): 1419.01 Peak output token throughput (tok/s): 19620.00 Peak concurrent requests: 105 Total token throughput (tok/s): 1641.02 Concurrency: 99.69 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 35888.57 Median E2E Latency (ms): 35321.48 P90 E2E Latency (ms): 41017.37 P99 E2E Latency (ms): 60343.22 ---------------Time to First Token---------------- Mean TTFT (ms): 35096.32 Median TTFT (ms): 34301.37 P99 TTFT (ms): 59966.25 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.63 Median TPOT (ms): 1.45 P99 TPOT (ms): 10.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.58 Median ITL (ms): 0.12 P95 ITL (ms): 0.23 P99 ITL (ms): 0.77 Max ITL (ms): 2086.12 ================================================== ``` # Laguna-M.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-M.1 Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4. ## Deployment Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model ships custom config code on the Hub, so `--trust-remote-code` is required (it is included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a # tagged release — install from main. The serving runtime is in the base dependencies, no extra needed: git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — `lmsysorg/sglang:latest` bundles the CUDA-13 runtime and the M.1 code. ```bash Command theme={null} # CUDA 13 — covers H200 + all Blackwell: docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0. **Key Features:** * **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing. * **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)). * **Long context**: 262,144 tokens, RoPE with YaRN. * **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0. * **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`. **Available Quantizations:**
Quantization Hugging Face path
BF16 [`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)
FP8 [`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)
NVFP4 [`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)
**License:** Apache 2.0 **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai). ## 2. Configuration Tips * **Trust remote code** (`--trust-remote-code`): Laguna-M.1 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model. * **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`. * **FP8**: On **Blackwell** the recipe adds `--fp8-gemm-backend triton` — the compressed-tensors block-FP8 weight scales aren't UE8M0-packed, so the default DeepGEMM path emits garbage on Blackwell (sm\_100); the Triton backend is correct (\~19% slower). Temporary workaround pending [PR #28662](https://github.com/sgl-project/sglang/pull/28662) (which fixes the scales and restores the DeepGEMM fast path). On **Hopper (H200)** FP8 uses DeepGEMM with no extra flag — pre-warm its multi-session JIT with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on each restart. * **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`). * **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. * **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched. * **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client. ## 3. Advanced Usage ### 3.1 Reasoning Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` ```text Output theme={null} =============== Reasoning =============== Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be tricky, but let me think. I remember that "percent" means per hundred, right? So 15% is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12 (half of that), so 24 + 12 = 36. [… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100, 1% × 15, and the fraction 3/20 × 240 — all give 36 …] So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36. =============== Answer ================== To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240: **240 × 0.15 = 36**. **Step-by-Step Explanation:** 1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15. 2. **Multiply by 240:** - Break it down: - 10% of 240 = 24 (since 240 × 0.1 = 24). - 5% of 240 = 12 (half of 24). - Add them: 24 + 12 = **36**. **Answer:** 15% of 240 is **36**. ``` Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly exceeds 3k tokens). The trace above is abbreviated; the model emits it in full. ### 3.2 Tool Calling Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` ```text Output theme={null} Tool: get_weather Args: {"location": "Beijing"} ``` ### 3.3 Prefill-Decode (PD) Disaggregation [PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. Laguna-M.1 is **global-attention with a standard KV cache** (no sliding window, no sparse "index" side-buffer), so its KV pages transfer with **no model-specific flags** — just the `--disaggregation-*` knobs. Both roles auto-select the same attention backend (FlashAttention-3) and page size because they share the model and flags, so the KV layout lines up for transfer. **Supported / validated topology:** * **Equal tensor parallelism** — prefill and decode run the same `--tp`. * **Single pipeline stage** — PP = 1 (the default). * **mooncake or NIXL** transfer backend over RDMA / InfiniBand. * Validated on **2 × 8×H200** (TP8 prefill + TP8 decode, BF16), one node each, over an 8× 400 Gb/s NDR InfiniBand fabric. Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Point `--disaggregation-ib-device` at your RDMA NIC(s). ```bash Prefill server (node A) theme={null} sglang serve \ --model-path poolside/Laguna-M.1 \ --trust-remote-code \ --reasoning-parser poolside_v1 \ --tool-call-parser poolside_v1 \ --tp 8 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30000 \ --disaggregation-bootstrap-port 8998 ``` ```bash Decode server (node B) theme={null} sglang serve \ --model-path poolside/Laguna-M.1 \ --trust-remote-code \ --reasoning-parser poolside_v1 \ --tool-call-parser poolside_v1 \ --tp 8 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \ --host 0.0.0.0 --port 30001 ``` Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint: ```bash PD router theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:30000 8998 \ --decode http://:30001 \ --policy round_robin \ --host 0.0.0.0 --port 8000 ``` Clients hit the router exactly like a single server — it splits each request across the two stages transparently: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://:8000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-M.1", messages=[{"role": "user", "content": "What is 2 + 2?"}], max_tokens=64, ) print(response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} 2 + 2 = 4 ``` **Transfer backend — mooncake (recommended).** mooncake honors `--disaggregation-ib-device` and establishes its RDMA connection at registration, so the **first request is already fast** (no cold start). It works with a single NIC or all eight; using **all 8 NICs lowers TTFT** (more aggregate bandwidth for the KV payload — the gap widens at longer context). On 8×H200 (random isl=512 / osl=256, 16 concurrent) it served ≈ **717 tok/s** output (≈ 2.2k tok/s total), mean **TTFT 244 ms**, mean **TPOT 17.7 ms**; with a single `mlx5_0` NIC, ≈ 697 tok/s and TTFT 287 ms (TPOT unchanged — decode is compute-bound). **Transfer backend — NIXL (works, with two caveats).** The NIXL path **ignores `--disaggregation-ib-device`** — that flag is mooncake-only. NIXL uses its UCX backend, whose NIC is selected by the **`UCX_NET_DEVICES`** environment variable. **Set it** (e.g. `export UCX_NET_DEVICES=mlx5_0:1`) on both servers; without it UCX cannot establish a working cross-node path and every KV transfer hangs until it hits the 300 s timeout (`Request … timed out … in KVPoll.WaitingForInput`) and returns a 500. With `UCX_NET_DEVICES` pinned, NIXL matches mooncake on quality and steady-state speed (≈ 720 tok/s, TTFT 230 ms, TPOT 17.7 ms). One difference: the **first request after launch pays a \~38 s one-time UCX connection cold-start** (a single port or all eight behave the same). Warm the path with one throwaway request after startup, or raise `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (default 300 s) so the first real request isn't dropped while UCX connects. **Validation.** PD disaggregation preserves output quality — disaggregated output matches non-disaggregated serving, and GSM8K (no-thinking, 200-question subset via the router) scored **0.945** (mooncake, 8 NICs) / **0.940** (NIXL) / **0.950** (mooncake, 1 NIC), all with 100% stop-rate and 0% errors — in line with single-node BF16 (≈ 0.93 on the full split). Logs confirm the split: the prefill node logs `Prefill batch` (CUDA graph off), the decode node logs `Decode batch` (CUDA graph on). # Laguna-S-2.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-S-2.1 Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4. ## Deployment Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points: * **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents. * **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves. On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only). ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**\~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family. **Key Features:** * **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert. * **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales. * **Long context**: 1,048,576 tokens. * **DFlash drafts**: matched draft models ship per quantization for low-latency serving. * **Hybrid reasoning**: `…` toggled per request via `chat_template_kwargs={"enable_thinking": …}`. **Available quantizations:**
Precision Target model Draft model
BF16 [`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1) [`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)
FP8 [`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8) [`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)
NVFP4 [`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4) [`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)
INT4 [`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4) [`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)
The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length). **License:** [OpenMDW-1.1](https://openmdw.ai/) **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai) ## 2. Configuration Tips **Attention backend** Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default. **BF16 memory on H200** BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic. **FP8 shared expert** `SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific. **FP8 and NVFP4 DFlash drafts** Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config. **DFlash memory** Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above). **BF16 reasoning length** BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window). **Chat template** On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) pass `--chat-template /chat_template.jinja` explicitly. **Thinking** Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored. **Served model id** The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id). ## 3. Advanced Usage ### 3.1 DFlash Speculative Decoding DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step. Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision. ### 3.2 Reasoning Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-S-2.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` Give generous `max_tokens` when thinking is enabled — hard problems regularly reason for thousands of tokens. Keep thinking off for short-form tasks. ### 3.3 Tool Calling Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-S-2.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` # Laguna-XS-2.1 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS-2.1 Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4. ## Deployment Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands). ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points: * **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents. * **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves. On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction [Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**\~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1). **Key Features:** * **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing. * **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads. * **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers). * **DFlash drafts**: matched draft models (5-layer, \~0.9 GB) ship per quantization for low-latency serving. * **Hybrid reasoning**: `…` toggled per request via `chat_template_kwargs={"enable_thinking": …}`. **Available quantizations:**
Precision Target model Draft model
BF16 [`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1) [`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)
FP8 [`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8) [`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)
NVFP4 [`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4) [`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)
INT4 [`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4) [`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)
The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length). **License:** Apache 2.0 **Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai). ## 2. Configuration Tips **Attention backend** Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%). **Quantized checkpoints cap plain TP at 4** `moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 \< 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200). **DFlash memory** Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic. **INT4 is mixed-precision** The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`. **Chat template** On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template /chat_template.jinja` explicitly there. **Thinking** Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored. **Served model id** The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id). ## 3. Advanced Usage ### 3.1 DFlash Speculative Decoding DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step: * Measured \~6 tokens/step at `tp 1`, \~4 at `tp 4` (greedy GSM8K, matched-precision pairs; \~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense. * Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. * The generated commands always pair the draft calibrated for the selected target precision. ### 3.2 Reasoning Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="poolside/Laguna-XS-2.1", messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}], max_tokens=2048, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) message = response.choices[0].message print("=============== Reasoning ===============") print(message.reasoning_content) print("=============== Answer ==================") print(message.content) ``` XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled (hard problems regularly reason for thousands of tokens), and keep thinking off for short-form tasks. ### 3.3 Tool Calling Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default). ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] response = client.chat.completions.create( model="poolside/Laguna-XS-2.1", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) message = response.choices[0].message if message.tool_calls: for call in message.tool_calls: print(f"Tool: {call.function.name}") print(f"Args: {call.function.arguments}") ``` # Laguna-XS.2 Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS.2 ## 1. Model Introduction [Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is an open-source hybrid sliding-window-attention MoE model from [Poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering work. **Key Features:** * **MoE**: 33.4B total parameters, 3.0B active per token, 256 routed experts (top-8) plus 1 shared. * **Long context**: 131,072 tokens. * **Agentic coding**: Tuned for tool-using software engineering agents and long-horizon execution. * **Hybrid reasoning**: `...` segments toggled per request via `chat_template_kwargs={"enable_thinking": ...}`. **Available Quantizations:**
Variant Hugging Face path
BF16 [`poolside/Laguna-XS.2`](https://huggingface.co/poolside/Laguna-XS.2)
FP8 [`poolside/Laguna-XS.2-FP8`](https://huggingface.co/poolside/Laguna-XS.2-FP8)
NVFP4 [`poolside/Laguna-XS.2-NVFP4`](https://huggingface.co/poolside/Laguna-XS.2-NVFP4)
**License:** Apache 2.0 For details, see the [Hugging Face model card](https://huggingface.co/poolside/Laguna-XS.2) and the [Laguna deeper-dive blog post](https://poolside.ai/blog/laguna-a-deeper-dive). ## 2. SGLang Installation Laguna-XS.2 support is on `main` but not yet in a tagged release; install from the SGLang nightly wheel index, or pull a pre-built Docker image: ```bash Command theme={null} # Install SGLang via pip (CUDA 13) — requires Python 3.10 (nightly wheels are cp310 only) python3 -m pip install --upgrade pip python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu130 \ "sglang[all]==0.5.12.dev20260509+g096ad02b0" # CUDA 12: swap to the cu129 index python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu129 \ "sglang[all]==0.5.12.dev20260509+g096ad02b0" # Or use Docker (multi-arch amd64/arm64; CUDA 13, H200 / B200) docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). ## 3. Model Deployment ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to generate a launch command for your hardware. ### 3.2 Configuration Tips * **Trust remote code** (`--trust-remote-code`): Laguna-XS.2 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model. * **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (\~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart. * **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `...` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `` tags in `content`. * **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments. * **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--dp --enable-dp-attention` with `--dp` matching `--tp` (tune independently if needed). * **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. ## 4. Model Invocation The samples below assume the server is reachable at `http://localhost:30000/v1`. ### 4.1 Basic Chat ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[ {"role": "user", "content": "What is the difference between TCP and UDP?"} ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two core protocols of the Internet Protocol (IP) suite, both used for network communication but with key differences: ## Connection Handling - **TCP**: Connection-oriented protocol that establishes a connection before data transfer (like a phone call) - **UDP**: Connectionless protocol that sends data without establishing a connection (like sending a letter) ## Reliability - **TCP**: Guaranteed delivery with error checking, retransmission of lost packets, and flow control - **UDP**: No guarantee of delivery; packets may be lost, duplicated, or arrive out of order ## Speed & Overhead - **TCP**: Slower due to connection setup, acknowledgment overhead, and error correction mechanisms - **UDP**: Faster with minimal overhead since it doesn't wait for acknowledgments or retransmit lost data ## Use Cases - **TCP**: Web browsing (HTTP/HTTPS), email (SMTP), file transfers (FTP), database connections - **UDP**: Video streaming, online gaming, VoIP calls, DNS queries, live broadcasts In essence, TCP prioritizes reliability over speed, while UDP prioritizes speed over reliability. ``` ### 4.2 Reasoning (Thinking Mode) Laguna-XS.2 emits reasoning between `...` tags. The `--reasoning-parser poolside_v1` flag separates the thinking text into `reasoning_content` so `content` holds only the final answer. Thinking is opt-in per request: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[ {"role": "user", "content": "If a train travels at 60 km/h for 2.5 hours, how far does it go?"} ], max_tokens=4096, extra_body={"chat_template_kwargs": {"enable_thinking": True}}, ) print("====== Reasoning Content ======") print(resp.choices[0].message.reasoning_content) print("====== Answer ======") print(resp.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content ====== The user is asking a straightforward math problem about distance, speed, and time. I need to calculate the distance using the formula: Distance = Speed × Time Given: - Speed = 60 km/h - Time = 2.5 hours So the calculation would be: Distance = 60 × 2.5 = 150 km This is a simple multiplication problem. I should provide a clear, direct answer and maybe explain the calculation briefly. ====== Answer ====== To find the distance, use the formula: Distance = Speed × Time Distance = 60 km/h × 2.5 h = 150 km The train travels **150 kilometers**. ``` To disable thinking, omit `extra_body` (off by default) or pass `chat_template_kwargs={"enable_thinking": False}` explicitly. ### 4.3 Tool Calling ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY", ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, }, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="poolside/Laguna-XS.2", messages=[{"role": "user", "content": "What's the weather in Tokyo?"}], tools=tools, ) msg = resp.choices[0].message print("====== Reasoning Content ======") print(msg.reasoning_content) print("====== Content ======") print(msg.content) print("====== Tool Calls ======") for tc in msg.tool_calls or []: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} ====== Reasoning Content ====== None ====== Content ====== I'll check the current weather in Tokyo for you. ====== Tool Calls ====== Function: get_weather Arguments: {"location": "Tokyo"} ``` `reasoning_content` is `None` because thinking is off by default; `content` carries the brief assistant message that precedes the tool call. Add `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` if you want interleaved reasoning before the tool call. ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: NVIDIA H200 (4×H200) * Model: `poolside/Laguna-XS.2` (BF16) * Tensor Parallelism: 4 * SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs) * Reasoning Parser: `poolside_v1` * Tool Call Parser: `poolside_v1` * Sampling: `temperature=0.6`, `max_tokens=16384`, `chat_template_kwargs={"enable_thinking": true}`, `n_repeats=1` * Grader: NeMo-Skills `math_verify` (math) and `eval_mcq` (multichoice) **Results (from [PR #24204](https://github.com/sgl-project/sglang/pull/24204)):** | Eval | Accuracy | | ------------------ | -------: | | GPQA Diamond | 0.5556 | | AIME 25 | 0.5667 | | MMLU | 0.836 | | SWE-Bench Verified | 0.6540 | ### 5.2 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 (1×H200 for TP=1, 4×H200 for TP=4) * Model: `poolside/Laguna-XS.2` (BF16) * SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs) * Workload: `sglang.bench_serving --backend sglang --dataset-name random` (defaults: `--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.0`) * Server flags identical to the accuracy runs above. #### 5.2.1 Latency Benchmark (10 prompts, concurrency = 1) ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 10 --max-concurrency 1 ``` | Metric | TP=1 | TP=4 | | ------------------------------- | -----: | -----: | | Successful requests | 10 | 10 | | Output token throughput (tok/s) | 193.10 | 238.88 | | Total token throughput (tok/s) | 471.82 | 583.68 | | Mean TTFT (ms) | 35.32 | 24.17 | | Mean TPOT (ms) | 5.10 | 4.13 | | Median ITL (ms) | 5.14 | 4.14 | #### 5.2.2 Throughput Benchmark (1000 prompts, concurrency = 100) ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang \ --host 0.0.0.0 --port 30000 \ --dataset-name random --num-prompts 1000 --max-concurrency 100 ``` | Metric | TP=1 | TP=4 | | ------------------------------------ | ------: | -------: | | Successful requests | 1000 | 1000 | | Request throughput (req/s) | 7.32 | 14.61 | | Output token throughput (tok/s) | 3739.30 | 7465.18 | | Peak output token throughput (tok/s) | 4718.00 | 10133.00 | | Total token throughput (tok/s) | 7485.82 | 14944.81 | | Mean TTFT (ms) | 115.17 | 68.36 | | Mean TPOT (ms) | 25.51 | 12.71 | | Median ITL (ms) | 21.31 | 10.64 | TP=4 delivers roughly 2.0× total-token throughput and \~1.7× lower mean TTFT compared to TP=1 on the `cc=100` random workload. # Qwen2.5-VL Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen2.5-VL ## 1. Model Introduction **[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing. **Key Features:** * **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images. * **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use. * **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments. * **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes. * **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc. * **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments. * **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs. * **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified). For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X as well as Intel Xeon CPU hardware platforms and different use cases. ### 3.1 Basic Configuration The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs as well as Intel Xeon CPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size. ### 3.2 Configuration Tips * **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed. * **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X. * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Qwen2.5-VL supports image inputs. Here's a basic example with single image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 2.31s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** Qwen2.5-VL can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 13.79s Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area. ``` **Note:** * You can also provide local file paths using `file://` protocol. * For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: Qwen2.5-VL-72B-Instruct * Tensor Parallelism: 8 * SGLang Version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 37.99 Total input tokens: 24781 Total input text tokens: 821 Total input vision tokens: 23960 Total generated tokens: 4220 Total generated tokens (retokenized): 2365 Request throughput (req/s): 0.26 Input token throughput (tok/s): 652.26 Output token throughput (tok/s): 111.07 Peak output token throughput (tok/s): 128.00 Peak concurrent requests: 2 Total token throughput (tok/s): 763.34 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3797.61 Median E2E Latency (ms): 3140.90 P90 E2E Latency (ms): 6545.54 P99 E2E Latency (ms): 7939.56 ---------------Time to First Token---------------- Mean TTFT (ms): 504.45 Median TTFT (ms): 510.93 P99 TTFT (ms): 521.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.82 Median TPOT (ms): 7.82 P99 TPOT (ms): 7.84 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.07 Median ITL (ms): 7.90 P95 ITL (ms): 15.79 P99 ITL (ms): 15.93 Max ITL (ms): 23.60 ================================================== ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen2.5-VL-72B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 454.68 Total input tokens: 2481865 Total input text tokens: 85865 Total input vision tokens: 2396000 Total generated tokens: 510855 Total generated tokens (retokenized): 296466 Request throughput (req/s): 2.20 Input token throughput (tok/s): 5458.50 Output token throughput (tok/s): 1123.55 Peak output token throughput (tok/s): 5004.00 Peak concurrent requests: 106 Total token throughput (tok/s): 6582.05 Concurrency: 98.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 44844.92 Median E2E Latency (ms): 42866.15 P90 E2E Latency (ms): 82798.20 P99 E2E Latency (ms): 106306.30 ---------------Time to First Token---------------- Mean TTFT (ms): 4507.79 Median TTFT (ms): 1180.83 P99 TTFT (ms): 39975.22 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 80.26 Median TPOT (ms): 82.38 P99 TPOT (ms): 152.89 ---------------Inter-Token Latency---------------- Mean ITL (ms): 100.66 Median ITL (ms): 13.26 P95 ITL (ms): 428.45 P99 ITL (ms): 1393.35 Max ITL (ms): 31943.26 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset: * Benchmark Command: ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` ```text Output theme={null} Benchmark time: 97.75084622902796 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.633, 'num': 30}, 'Agriculture': {'acc': 0.5, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.367, 'num': 30}, 'Art': {'acc': 0.767, 'num': 30}, 'Art_Theory': {'acc': 0.9, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.7, 'num': 30}, 'Biology': {'acc': 0.467, 'num': 30}, 'Chemistry': {'acc': 0.433, 'num': 30}, 'Clinical_Medicine': {'acc': 0.733, 'num': 30}, 'Computer_Science': {'acc': 0.567, 'num': 30}, 'Design': {'acc': 0.833, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30}, 'Economics': {'acc': 0.767, 'num': 30}, 'Electronics': {'acc': 0.433, 'num': 30}, 'Energy_and_Power': {'acc': 0.467, 'num': 30}, 'Finance': {'acc': 0.533, 'num': 30}, 'Geography': {'acc': 0.633, 'num': 30}, 'History': {'acc': 0.7, 'num': 30}, 'Literature': {'acc': 0.867, 'num': 30}, 'Manage': {'acc': 0.633, 'num': 30}, 'Marketing': {'acc': 0.733, 'num': 30}, 'Materials': {'acc': 0.333, 'num': 30}, 'Math': {'acc': 0.533, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.433, 'num': 30}, 'Music': {'acc': 0.367, 'num': 30}, 'Overall': {'acc': 0.62, 'num': 900}, 'Overall-Art and Design': {'acc': 0.717, 'num': 120}, 'Overall-Business': {'acc': 0.66, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.693, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120}, 'Overall-Science': {'acc': 0.553, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210}, 'Pharmacy': {'acc': 0.833, 'num': 30}, 'Physics': {'acc': 0.7, 'num': 30}, 'Psychology': {'acc': 0.767, 'num': 30}, 'Public_Health': {'acc': 0.733, 'num': 30}, 'Sociology': {'acc': 0.767, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.62 ``` # Qwen3 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3 ## 1. Model Introduction [Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications. This generation delivers comprehensive upgrades across the board: * **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. * **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages. * **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation. * **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts. * **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance. * **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions. For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Memory Management:** Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K. * **Expert Parallelism:** SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details. * **Kernel Tuning:** For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton). * **Speculative Decoding:** Using Speculative Decoding for latency-sensitive scenarios. * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm * `--speculative-num-steps 3`: Number of speculative verification rounds * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens * `--speculative-num-draft-tokens 4`: Number of draft tokens per step * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan). * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Thinking-2507 \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method. ... So conclusion: The answer is 36. =============== Content ================= To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution: .... ### Final Answer: $$ \boxed{36} $$ Thus, 15% of 240 is **36**. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Qwen3 supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Thinking-2507 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them. =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-235B-A22B-Thinking-2507", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞" ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Qwen3-235B-A22B-Instruct-2507 * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --tp 8 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 43.56 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4206 Request throughput (req/s): 0.23 Input token throughput (tok/s): 140.07 Output token throughput (tok/s): 96.65 Peak output token throughput (tok/s): 100.00 Peak concurrent requests: 2 Total token throughput (tok/s): 236.72 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4353.63 Median E2E Latency (ms): 3475.79 ---------------Time to First Token---------------- Mean TTFT (ms): 99.03 Median TTFT (ms): 92.18 P99 TTFT (ms): 166.05 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.12 Median TPOT (ms): 10.12 P99 TPOT (ms): 10.15 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.13 Median ITL (ms): 10.12 P95 ITL (ms): 10.49 P99 ITL (ms): 10.70 Max ITL (ms): 13.45 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 48.95 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 40725 Total generated tokens (retokenized): 40716 Request throughput (req/s): 1.63 Input token throughput (tok/s): 810.44 Output token throughput (tok/s): 832.04 Peak output token throughput (tok/s): 1151.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1642.48 Concurrency: 13.61 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8326.72 Median E2E Latency (ms): 8827.86 ---------------Time to First Token---------------- Mean TTFT (ms): 215.70 Median TTFT (ms): 88.82 P99 TTFT (ms): 727.08 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.36 Median TPOT (ms): 16.12 P99 TPOT (ms): 24.09 ---------------Inter-Token Latency---------------- Mean ITL (ms): 15.96 Median ITL (ms): 14.52 P95 ITL (ms): 16.04 P99 ITL (ms): 67.69 Max ITL (ms): 457.52 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 92.07 Total input tokens: 249831 Total input text tokens: 249831 Total input vision tokens: 0 Total generated tokens: 252162 Total generated tokens (retokenized): 251124 Request throughput (req/s): 5.43 Input token throughput (tok/s): 2713.46 Output token throughput (tok/s): 2738.78 Peak output token throughput (tok/s): 4400.00 Peak concurrent requests: 110 Total token throughput (tok/s): 5452.24 Concurrency: 90.50 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16665.09 Median E2E Latency (ms): 16060.10 ---------------Time to First Token---------------- Mean TTFT (ms): 260.55 Median TTFT (ms): 122.68 P99 TTFT (ms): 863.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.94 Median TPOT (ms): 34.04 P99 TPOT (ms): 41.19 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.59 Median ITL (ms): 23.54 P95 ITL (ms): 69.79 P99 ITL (ms): 119.09 Max ITL (ms): 577.70 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --tp 8 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 457.45 Total input tokens: 6101 Total input text tokens: 6101 Total input vision tokens: 0 Total generated tokens: 44452 Total generated tokens (retokenized): 44059 Request throughput (req/s): 0.02 Input token throughput (tok/s): 13.34 Output token throughput (tok/s): 97.17 Peak output token throughput (tok/s): 100.00 Peak concurrent requests: 2 Total token throughput (tok/s): 110.51 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 45742.42 Median E2E Latency (ms): 49266.87 ---------------Time to First Token---------------- Mean TTFT (ms): 110.60 Median TTFT (ms): 109.36 P99 TTFT (ms): 167.43 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.23 Median TPOT (ms): 10.24 P99 TPOT (ms): 10.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.27 Median ITL (ms): 10.26 P95 ITL (ms): 10.71 P99 ITL (ms): 10.97 Max ITL (ms): 15.79 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 340.17 Total input tokens: 39668 Total input text tokens: 39668 Total input vision tokens: 0 Total generated tokens: 318226 Total generated tokens (retokenized): 318104 Request throughput (req/s): 0.24 Input token throughput (tok/s): 116.61 Output token throughput (tok/s): 935.49 Peak output token throughput (tok/s): 1120.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1052.10 Concurrency: 13.85 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 58885.30 Median E2E Latency (ms): 59238.70 ---------------Time to First Token---------------- Mean TTFT (ms): 169.71 Median TTFT (ms): 101.61 P99 TTFT (ms): 455.71 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.82 Median TPOT (ms): 14.91 P99 TPOT (ms): 15.20 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.76 Median ITL (ms): 14.63 P95 ITL (ms): 15.46 P99 ITL (ms): 16.62 Max ITL (ms): 104.94 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 544.83 Total input tokens: 158939 Total input text tokens: 158939 Total input vision tokens: 0 Total generated tokens: 1300705 Total generated tokens (retokenized): 1293015 Request throughput (req/s): 0.59 Input token throughput (tok/s): 291.72 Output token throughput (tok/s): 2387.34 Peak output token throughput (tok/s): 3008.00 Peak concurrent requests: 68 Total token throughput (tok/s): 2679.06 Concurrency: 56.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 95937.70 Median E2E Latency (ms): 99362.32 ---------------Time to First Token---------------- Mean TTFT (ms): 265.03 Median TTFT (ms): 129.11 P99 TTFT (ms): 823.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 23.66 Median TPOT (ms): 24.07 P99 TPOT (ms): 24.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 23.54 Median ITL (ms): 23.07 P95 ITL (ms): 25.92 P99 ITL (ms): 63.87 Max ITL (ms): 408.30 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 44.82 Total input tokens: 41941 Total input text tokens: 41941 Total input vision tokens: 0 Total generated tokens: 4210 Total generated tokens (retokenized): 4210 Request throughput (req/s): 0.22 Input token throughput (tok/s): 935.86 Output token throughput (tok/s): 93.94 Peak output token throughput (tok/s): 99.00 Peak concurrent requests: 2 Total token throughput (tok/s): 1029.80 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4479.60 Median E2E Latency (ms): 3622.99 ---------------Time to First Token---------------- Mean TTFT (ms): 139.90 Median TTFT (ms): 114.85 P99 TTFT (ms): 225.17 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.31 Median TPOT (ms): 10.33 P99 TPOT (ms): 10.51 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.33 Median ITL (ms): 10.33 P95 ITL (ms): 10.73 P99 ITL (ms): 10.93 Max ITL (ms): 14.48 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 50.68 Total input tokens: 300020 Total input text tokens: 300020 Total input vision tokens: 0 Total generated tokens: 41589 Total generated tokens (retokenized): 41578 Request throughput (req/s): 1.58 Input token throughput (tok/s): 5920.41 Output token throughput (tok/s): 820.69 Peak output token throughput (tok/s): 1200.00 Peak concurrent requests: 20 Total token throughput (tok/s): 6741.10 Concurrency: 13.90 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8805.54 Median E2E Latency (ms): 9368.79 ---------------Time to First Token---------------- Mean TTFT (ms): 284.29 Median TTFT (ms): 168.48 P99 TTFT (ms): 1027.21 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.81 Median TPOT (ms): 16.66 P99 TPOT (ms): 27.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.42 Median ITL (ms): 13.68 P95 ITL (ms): 17.23 P99 ITL (ms): 90.75 Max ITL (ms): 574.64 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-235B-A22B-Instruct-2507 \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 94.77 Total input tokens: 1273893 Total input text tokens: 1273893 Total input vision tokens: 0 Total generated tokens: 169680 Total generated tokens (retokenized): 169640 Request throughput (req/s): 3.38 Input token throughput (tok/s): 13441.86 Output token throughput (tok/s): 1790.43 Peak output token throughput (tok/s): 2687.00 Peak concurrent requests: 70 Total token throughput (tok/s): 15232.28 Concurrency: 58.63 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 17364.14 Median E2E Latency (ms): 17495.95 ---------------Time to First Token---------------- Mean TTFT (ms): 238.22 Median TTFT (ms): 203.27 P99 TTFT (ms): 510.48 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 32.50 Median TPOT (ms): 34.27 P99 TPOT (ms): 40.59 ---------------Inter-Token Latency---------------- Mean ITL (ms): 32.36 Median ITL (ms): 22.50 P95 ITL (ms): 97.81 P99 ITL (ms): 151.55 Max ITL (ms): 352.79 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * **Results**: * Qwen/Qwen3-235B-A22B-Instruct-2507 ```text Output theme={null} Accuracy: 0.945 Invalid: 0.000 Latency: 11.980 s Output throughput: 2358.105 token/s ``` # Qwen3-Coder Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder ## 1. Model Introduction [Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks. **Key Features:** * **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks. * **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services. * **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents. * **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages. * **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio. * **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified). * **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified). For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms. ### 3.1 Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method. ### 3.2 Configuration Tips **AMD (MI300X/MI325X/MI355X):** * **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory. * **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement. * **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage. * **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`. **NVIDIA (B200/GB200):** * **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above. * **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`). * **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput. **Intel Xeon CPU:** * Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. **General:** * **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Code Generation Example ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints." } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-480B-A35B-Instruct", messages=messages, max_tokens=2048, temperature=0.7 ) print(response.choices[0].message.content) ``` **Example Output:** ````text Output theme={null} ```python from typing import List, Optional, TypeVar T = TypeVar('T') def binary_search(arr: List[T], target: T) -> Optional[int]: """ Perform binary search on a sorted list to find the index of a target element. This function implements the binary search algorithm, which efficiently finds a target value in a sorted array by repeatedly dividing the search interval in half. Args: arr (List[T]): A sorted list of elements to search through. target (T): The element to search for in the list. Returns: Optional[int]: The index of the target element if found, None otherwise. Time Complexity: O(log n) where n is the number of elements in the array. Space Complexity: O(1) - iterative implementation uses constant extra space. Examples: >>> binary_search([1, 2, 3, 4, 5], 3) 2 >>> binary_search([1, 2, 3, 4, 5], 6) None >>> binary_search(['a', 'b', 'c', 'd'], 'b') 1 >>> binary_search([], 1) None """ if not arr: return None left: int = 0 right: int = len(arr) - 1 while left <= right: mid: int = (left + right) // 2 if arr[mid] == target: return mid elif arr[mid] < target: left = mid + 1 else: right = mid - 1 return None # Alternative recursive implementation def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]: """ Perform binary search recursively on a sorted list to find the index of a target element. Args: arr (List[T]): A sorted list of elements to search through. target (T): The element to search for in the list. left (int): Left boundary of the search range (inclusive). right (Optional[int]): Right boundary of the search range (inclusive). Returns: Optional[int]: The index of the target element if found, None otherwise. Time Complexity: O(log n) where n is the number of elements in the array. Space Complexity: O(log n) due to recursive call stack. Examples: >>> binary_search_recursive([1, 2, 3, 4, 5], 3) 2 >>> binary_search_recursive([1, 2, 3, 4, 5], 6) None """ if not arr: return None if right is None: right = len(arr) - 1 if left > right: return None mid: int = (left + right) // 2 if arr[mid] == target: return mid elif arr[mid] < target: return binary_search_recursive(arr, target, mid + 1, right) else: return binary_search_recursive(arr, target, left, mid - 1) ``` This implementation provides: 1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient 2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes 3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type 4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples 5. **Edge case handling**: Empty lists, elements not found, etc. 6. **Clear variable names**: Self-documenting code 7. **Examples**: Doctest-style examples in the docstring The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found. ```` #### 4.2.2 Tool Calling Example Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model: ```shell Command theme={null} SGLANG_USE_AITER=0 python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-30B-A3B-Instruct \ --tp 1 \ --context-length 8192 \ --page-size 32 \ --tool-call-parser qwen3_coder ``` **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) # Define available tools tools = [ { "type": "function", "function": { "name": "execute_code", "description": "Execute Python code and return the result", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "The Python code to execute" } }, "required": ["code"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-30B-A3B-Instruct", messages=[ {"role": "user", "content": "Calculate the factorial of 10 using Python"} ], tools=tools, temperature=0.7 ) # Check if the model wants to call a tool if response.choices[0].message.tool_calls: tool_call = response.choices[0].message.tool_calls[0] print(f"Tool: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") else: # Model may return tool call in content format print(response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Tool: execute_code Arguments: {"code": "def factorial(n):\n if n == 0 or n == 1:\n return 1\n else:\n return n * factorial(n-1)\n\nresult = factorial(10)\nresult"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: AMD MI300X GPU (8x) * Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 * Tensor Parallelism: 8 * Expert Parallelism: 2 * sglang version: 0.5.7 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset. #### 5.1.1 AMD Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} SGLANG_USE_AITER=0 python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --tp 8 \ --ep 2 \ --context-length 8192 \ --page-size 32 \ --trust-remote-code ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 73.79 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4104 Request throughput (req/s): 0.14 Input token throughput (tok/s): 82.68 Output token throughput (tok/s): 57.19 Peak output token throughput (tok/s): 59.00 Peak concurrent requests: 2 Total token throughput (tok/s): 139.86 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7376.26 Median E2E Latency (ms): 5851.51 P90 E2E Latency (ms): 13351.89 P99 E2E Latency (ms): 16908.32 ---------------Time to First Token---------------- Mean TTFT (ms): 191.93 Median TTFT (ms): 126.06 P99 TTFT (ms): 662.15 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 17.06 Median TPOT (ms): 17.07 P99 TPOT (ms): 17.08 ---------------Inter-Token Latency---------------- Mean ITL (ms): 17.06 Median ITL (ms): 17.06 P95 ITL (ms): 17.14 P99 ITL (ms): 17.19 Max ITL (ms): 18.53 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 87.04 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40364 Request throughput (req/s): 0.92 Input token throughput (tok/s): 455.77 Output token throughput (tok/s): 468.83 Peak output token throughput (tok/s): 608.00 Peak concurrent requests: 20 Total token throughput (tok/s): 924.59 Concurrency: 13.76 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 14966.88 Median E2E Latency (ms): 15871.93 P90 E2E Latency (ms): 24983.41 P99 E2E Latency (ms): 29504.85 ---------------Time to First Token---------------- Mean TTFT (ms): 388.94 Median TTFT (ms): 157.49 P99 TTFT (ms): 1318.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.41 Median TPOT (ms): 29.22 P99 TPOT (ms): 43.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 28.64 Median ITL (ms): 26.42 P95 ITL (ms): 27.51 P99 ITL (ms): 131.63 Max ITL (ms): 995.11 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 177.82 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 168387 Request throughput (req/s): 1.80 Input token throughput (tok/s): 893.84 Output token throughput (tok/s): 956.80 Peak output token throughput (tok/s): 1728.00 Peak concurrent requests: 70 Total token throughput (tok/s): 1850.64 Concurrency: 58.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 32716.53 Median E2E Latency (ms): 30896.37 P90 E2E Latency (ms): 65605.24 P99 E2E Latency (ms): 80970.63 ---------------Time to First Token---------------- Mean TTFT (ms): 372.97 Median TTFT (ms): 181.67 P99 TTFT (ms): 529.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 62.98 Median TPOT (ms): 50.44 P99 TPOT (ms): 204.24 ---------------Inter-Token Latency---------------- Mean ITL (ms): 60.95 Median ITL (ms): 37.87 P95 ITL (ms): 143.98 P99 ITL (ms): 148.02 Max ITL (ms): 36863.32 ================================================== ``` #### 5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark The following runs use the same random dataset benchmark client commands as the AMD section. On B200, launch the server with the following command: ````bash theme={null} sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code ##### 5.1.2.1 FP8 Model - Low Concurrency: ```text Output ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 42.68 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4204 Request throughput (req/s): 0.23 Input token throughput (tok/s): 142.95 Output token throughput (tok/s): 98.88 Peak output token throughput (tok/s): 102.00 Peak concurrent requests: 2 Total token throughput (tok/s): 241.83 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4266.06 Median E2E Latency (ms): 3420.24 P90 E2E Latency (ms): 7717.19 P99 E2E Latency (ms): 9504.50 ---------------Time to First Token---------------- Mean TTFT (ms): 112.03 Median TTFT (ms): 112.70 P99 TTFT (ms): 115.35 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.87 Median TPOT (ms): 9.86 P99 TPOT (ms): 9.92 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.87 Median ITL (ms): 9.87 P95 ITL (ms): 10.06 P99 ITL (ms): 10.18 Max ITL (ms): 14.80 ================================================== ```` * Medium Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 60.80 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40543 Request throughput (req/s): 1.32 Input token throughput (tok/s): 652.43 Output token throughput (tok/s): 671.13 Peak output token throughput (tok/s): 864.00 Peak concurrent requests: 20 Total token throughput (tok/s): 1323.57 Concurrency: 13.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 10587.26 Median E2E Latency (ms): 11486.18 P90 E2E Latency (ms): 17374.75 P99 E2E Latency (ms): 21107.18 ---------------Time to First Token---------------- Mean TTFT (ms): 155.27 Median TTFT (ms): 121.57 P99 TTFT (ms): 294.31 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 20.77 Median TPOT (ms): 21.13 P99 TPOT (ms): 23.62 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.49 Median ITL (ms): 18.73 P95 ITL (ms): 19.65 P99 ITL (ms): 98.85 Max ITL (ms): 536.87 ================================================== ``` * High Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 100.07 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 169119 Request throughput (req/s): 3.20 Input token throughput (tok/s): 1588.32 Output token throughput (tok/s): 1700.19 Peak output token throughput (tok/s): 2303.00 Peak concurrent requests: 71 Total token throughput (tok/s): 3288.51 Concurrency: 57.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 18114.01 Median E2E Latency (ms): 18279.15 P90 E2E Latency (ms): 30557.22 P99 E2E Latency (ms): 35889.84 ---------------Time to First Token---------------- Mean TTFT (ms): 346.40 Median TTFT (ms): 129.75 P99 TTFT (ms): 1370.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 33.76 Median TPOT (ms): 34.62 P99 TPOT (ms): 39.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 33.48 Median ITL (ms): 25.70 P95 ITL (ms): 99.36 P99 ITL (ms): 132.30 Max ITL (ms): 1132.39 ================================================== ``` ##### 5.1.2.2 NVFP4 Model * Low Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 34.49 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.29 Input token throughput (tok/s): 176.87 Output token throughput (tok/s): 122.34 Peak output token throughput (tok/s): 127.00 Peak concurrent requests: 2 Total token throughput (tok/s): 299.21 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3448.01 Median E2E Latency (ms): 2768.11 P90 E2E Latency (ms): 6225.73 P99 E2E Latency (ms): 7668.26 ---------------Time to First Token---------------- Mean TTFT (ms): 104.55 Median TTFT (ms): 105.38 P99 TTFT (ms): 105.63 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.94 Median TPOT (ms): 7.95 P99 TPOT (ms): 7.97 ---------------Inter-Token Latency---------------- Mean ITL (ms): 7.94 Median ITL (ms): 7.94 P95 ITL (ms): 8.05 P99 ITL (ms): 8.11 Max ITL (ms): 24.64 ================================================== ``` * Medium Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 43.30 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 39975 Request throughput (req/s): 1.85 Input token throughput (tok/s): 916.16 Output token throughput (tok/s): 942.42 Peak output token throughput (tok/s): 1264.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1858.57 Concurrency: 13.90 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7521.95 Median E2E Latency (ms): 8246.89 P90 E2E Latency (ms): 12370.93 P99 E2E Latency (ms): 15023.96 ---------------Time to First Token---------------- Mean TTFT (ms): 137.27 Median TTFT (ms): 109.59 P99 TTFT (ms): 208.78 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.69 Median TPOT (ms): 14.87 P99 TPOT (ms): 17.63 ---------------Inter-Token Latency---------------- Mean ITL (ms): 14.51 Median ITL (ms): 12.75 P95 ITL (ms): 13.33 P99 ITL (ms): 92.85 Max ITL (ms): 113.70 ================================================== ``` * High Concurrency: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 73.93 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 170134 Total generated tokens (retokenized): 168841 Request throughput (req/s): 4.33 Input token throughput (tok/s): 2149.98 Output token throughput (tok/s): 2301.42 Peak output token throughput (tok/s): 3497.00 Peak concurrent requests: 71 Total token throughput (tok/s): 4451.40 Concurrency: 58.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 13463.58 Median E2E Latency (ms): 13498.74 P90 E2E Latency (ms): 22957.10 P99 E2E Latency (ms): 26656.95 ---------------Time to First Token---------------- Mean TTFT (ms): 239.00 Median TTFT (ms): 113.42 P99 TTFT (ms): 713.87 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 25.13 Median TPOT (ms): 26.02 P99 TPOT (ms): 30.90 ---------------Inter-Token Latency---------------- Mean ITL (ms): 24.92 Median ITL (ms): 16.68 P95 ITL (ms): 93.33 P99 ITL (ms): 119.26 Max ITL (ms): 548.82 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` ##### AMD (MI300X/MI325X/MI355X) * **Results**: * Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 ``` Accuracy: 0.965 Invalid: 0.000 Latency: 23.084 s Output throughput: 1148.425 token/s ``` ##### NVIDIA (B200/GB200) For deployment commands, see [Section 3.1](#3-1-configuration). * Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 ``` Accuracy: 0.965 Invalid: 0.000 Latency: 14.870 s Output throughput: 1777.726 token/s ``` * nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4) ``` Accuracy: 0.960 Invalid: 0.000 Latency: 13.948 s Output throughput: 1988.548 token/s ``` # Qwen3-Coder-Next Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder-Next ## 1. Model Introduction [Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture. **Key Features:** * **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency. * **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser. * **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents. * **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale. * **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms. For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). **Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later. ## 3. Model Deployment This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. ### 3.2 Configuration Tips * **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`. * **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag. * **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed. * **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation **Deployment Command:** ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --tool-call-parser qwen3_coder \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Code Generation Example ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."} ], max_tokens=2048 ) print(response.choices[0].message.content) ``` **Example Output:** ````text Output theme={null} Here's a Python function implementing binary search on a sorted list, with comprehensive type hints: ```python from typing import Sequence, TypeVar, Optional T = TypeVar('T') def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]: """ Perform binary search on a sorted list to find the index of a target element. Args: sorted_list: A sequence (e.g., list, tuple) sorted in ascending order. target: The element to search for in the list. Returns: The index of the target element if found, or None if not found. Time Complexity: O(log n) Space Complexity: O(1) Note: The function assumes the list is sorted in ascending order. If the list contains duplicate elements, it returns the index of one of them. """ left = 0 right = len(sorted_list) - 1 while left <= right: mid = (left + right) // 2 mid_val = sorted_list[mid] if mid_val == target: return mid elif mid_val < target: left = mid + 1 else: right = mid - 1 return None ``` ### Example usage: ```python # Example 1: Finding an existing element numbers = [1, 3, 5, 7, 9, 11] print(binary_search(numbers, 7)) # Output: 3 # Example 2: Element not in the list print(binary_search(numbers, 4)) # Output: None # Example 3: Empty list print(binary_search([], 5)) # Output: None # Example 4: Single element print(binary_search([1], 1)) # Output: 0 print(binary_search([1], 2)) # Output: None ``` ### Key features: - Uses `TypeVar` to support generic types (as long as comparison operations are defined) - Returns `Optional[int]` to indicate either the index or no match found - Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.) - Includes comprehensive docstring with time/space complexity - Implements standard iterative binary search for O(1) space complexity ```` #### 4.2.2 Streaming Example ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."} ], max_tokens=512, stream=True ) for chunk in response: if chunk.choices and chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) print() ``` **Example Output:** ```text Output theme={null} A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal. ``` #### 4.2.3 Tool Calling Example Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above. **Python Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "execute_code", "description": "Execute Python code and return the result", "parameters": { "type": "object", "properties": { "code": { "type": "string", "description": "The Python code to execute" } }, "required": ["code"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3-Coder-Next", messages=[ {"role": "user", "content": "Calculate the factorial of 10 using Python"} ], tools=tools ) # Check if the model wants to call a tool if response.choices[0].message.tool_calls: tool_call = response.choices[0].message.tool_calls[0] print(f"Tool: {tool_call.function.name}") print(f"Arguments: {tool_call.function.arguments}") else: print(response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Tool: execute_code Arguments: {"code": "import math\nmath.factorial(10)"} ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (2x) * Model: Qwen/Qwen3-Coder-Next * Tensor Parallelism: 2 * sglang version: 0.5.8+ #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --host 0.0.0.0 \ --port 30000 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 27.86 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4218 Request throughput (req/s): 0.36 Input token throughput (tok/s): 219.00 Output token throughput (tok/s): 151.48 Peak output token throughput (tok/s): 166.00 Peak concurrent requests: 2 Total token throughput (tok/s): 370.48 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2784.14 Median E2E Latency (ms): 2258.08 P90 E2E Latency (ms): 5044.43 P99 E2E Latency (ms): 6130.52 ---------------Time to First Token---------------- Mean TTFT (ms): 161.68 Median TTFT (ms): 168.09 P99 TTFT (ms): 183.26 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.19 Median TPOT (ms): 6.23 P99 TPOT (ms): 6.32 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.23 Median ITL (ms): 6.23 P95 ITL (ms): 6.51 P99 ITL (ms): 6.64 Max ITL (ms): 13.45 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 39.06 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 40805 Total generated tokens (retokenized): 40789 Request throughput (req/s): 2.05 Input token throughput (tok/s): 1015.62 Output token throughput (tok/s): 1044.73 Peak output token throughput (tok/s): 1664.00 Peak concurrent requests: 21 Total token throughput (tok/s): 2060.34 Concurrency: 14.16 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 6910.97 Median E2E Latency (ms): 7248.27 P90 E2E Latency (ms): 11612.63 P99 E2E Latency (ms): 13933.91 ---------------Time to First Token---------------- Mean TTFT (ms): 183.48 Median TTFT (ms): 156.50 P99 TTFT (ms): 311.46 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 13.61 Median TPOT (ms): 13.59 P99 TPOT (ms): 21.11 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.22 Median ITL (ms): 9.76 P95 ITL (ms): 10.43 P99 ITL (ms): 158.04 Max ITL (ms): 394.39 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 102.81 Total input tokens: 249831 Total input text tokens: 249831 Total generated tokens: 252662 Total generated tokens (retokenized): 252536 Request throughput (req/s): 4.86 Input token throughput (tok/s): 2429.99 Output token throughput (tok/s): 2457.53 Peak output token throughput (tok/s): 5299.00 Peak concurrent requests: 109 Total token throughput (tok/s): 4887.52 Concurrency: 94.28 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 19385.20 Median E2E Latency (ms): 17584.09 P90 E2E Latency (ms): 36762.15 P99 E2E Latency (ms): 42518.35 ---------------Time to First Token---------------- Mean TTFT (ms): 270.62 Median TTFT (ms): 159.65 P99 TTFT (ms): 938.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 38.57 Median TPOT (ms): 41.78 P99 TPOT (ms): 53.28 ---------------Inter-Token Latency---------------- Mean ITL (ms): 37.90 Median ITL (ms): 18.26 P95 ITL (ms): 167.82 P99 ITL (ms): 311.45 Max ITL (ms): 993.20 ================================================== ``` #### 5.1.2 Reasoning Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Coder-Next \ --tp 2 \ --host 0.0.0.0 \ --port 30000 ``` ##### 5.1.2.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 285.02 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 44462 Total generated tokens (retokenized): 44432 Request throughput (req/s): 0.04 Input token throughput (tok/s): 21.41 Output token throughput (tok/s): 156.00 Peak output token throughput (tok/s): 173.00 Peak concurrent requests: 2 Total token throughput (tok/s): 177.40 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 28499.54 Median E2E Latency (ms): 30424.65 P90 E2E Latency (ms): 49132.26 P99 E2E Latency (ms): 51075.28 ---------------Time to First Token---------------- Mean TTFT (ms): 95.51 Median TTFT (ms): 93.86 P99 TTFT (ms): 112.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.24 Median TPOT (ms): 6.30 P99 TPOT (ms): 6.60 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.39 Median ITL (ms): 6.34 P95 ITL (ms): 7.16 P99 ITL (ms): 7.42 Max ITL (ms): 12.48 ================================================== ``` ##### 5.1.2.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 237.77 Total input tokens: 39668 Total input text tokens: 39668 Total generated tokens: 318306 Total generated tokens (retokenized): 315646 Request throughput (req/s): 0.34 Input token throughput (tok/s): 166.83 Output token throughput (tok/s): 1338.72 Peak output token throughput (tok/s): 1727.00 Peak concurrent requests: 19 Total token throughput (tok/s): 1505.55 Concurrency: 13.88 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 41266.21 Median E2E Latency (ms): 41010.10 P90 E2E Latency (ms): 77574.22 P99 E2E Latency (ms): 82688.04 ---------------Time to First Token---------------- Mean TTFT (ms): 140.73 Median TTFT (ms): 84.52 P99 TTFT (ms): 365.86 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.32 Median TPOT (ms): 10.38 P99 TPOT (ms): 10.87 ---------------Inter-Token Latency---------------- Mean ITL (ms): 10.34 Median ITL (ms): 10.19 P95 ITL (ms): 10.75 P99 ITL (ms): 11.18 Max ITL (ms): 206.79 ================================================== ``` ##### 5.1.2.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 8000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 384.82 Total input tokens: 158939 Total input text tokens: 158939 Total generated tokens: 1301025 Total generated tokens (retokenized): 1299908 Request throughput (req/s): 0.83 Input token throughput (tok/s): 413.02 Output token throughput (tok/s): 3380.83 Peak output token throughput (tok/s): 4317.00 Peak concurrent requests: 69 Total token throughput (tok/s): 3793.85 Concurrency: 56.42 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 67847.54 Median E2E Latency (ms): 70724.38 P90 E2E Latency (ms): 120888.83 P99 E2E Latency (ms): 133234.48 ---------------Time to First Token---------------- Mean TTFT (ms): 212.24 Median TTFT (ms): 115.96 P99 TTFT (ms): 652.93 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 16.76 Median TPOT (ms): 16.99 P99 TPOT (ms): 18.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 16.64 Median ITL (ms): 15.83 P95 ITL (ms): 31.64 P99 ITL (ms): 90.85 Max ITL (ms): 576.60 ================================================== ``` #### 5.1.3 Summarization Scenario Benchmark ##### 5.1.3.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 29.42 Total input tokens: 41941 Total input text tokens: 41941 Total generated tokens: 4220 Total generated tokens (retokenized): 4220 Request throughput (req/s): 0.34 Input token throughput (tok/s): 1425.35 Output token throughput (tok/s): 143.42 Peak output token throughput (tok/s): 169.00 Peak concurrent requests: 3 Total token throughput (tok/s): 1568.77 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2941.19 Median E2E Latency (ms): 2411.84 P90 E2E Latency (ms): 5661.26 P99 E2E Latency (ms): 6497.45 ---------------Time to First Token---------------- Mean TTFT (ms): 139.46 Median TTFT (ms): 160.33 P99 TTFT (ms): 184.30 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.56 Median TPOT (ms): 6.65 P99 TPOT (ms): 7.29 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.65 Median ITL (ms): 6.68 P95 ITL (ms): 7.39 P99 ITL (ms): 7.51 Max ITL (ms): 16.34 ================================================== ``` ##### 5.1.3.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 41.62 Total input tokens: 300020 Total input text tokens: 300020 Total generated tokens: 41669 Total generated tokens (retokenized): 41664 Request throughput (req/s): 1.92 Input token throughput (tok/s): 7208.67 Output token throughput (tok/s): 1001.19 Peak output token throughput (tok/s): 1536.00 Peak concurrent requests: 21 Total token throughput (tok/s): 8209.86 Concurrency: 14.27 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 7421.29 Median E2E Latency (ms): 7985.77 P90 E2E Latency (ms): 12122.09 P99 E2E Latency (ms): 14595.05 ---------------Time to First Token---------------- Mean TTFT (ms): 248.49 Median TTFT (ms): 179.25 P99 TTFT (ms): 915.90 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 14.13 Median TPOT (ms): 14.28 P99 TPOT (ms): 24.02 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.80 Median ITL (ms): 10.46 P95 ITL (ms): 11.00 P99 ITL (ms): 173.14 Max ITL (ms): 823.32 ================================================== ``` ##### 5.1.3.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-Coder-Next \ --dataset-name random \ --random-input-len 8000 \ --random-output-len 1000 \ --num-prompts 320 \ --max-concurrency 64 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 320 Benchmark duration (s): 85.74 Total input tokens: 1273893 Total input text tokens: 1273893 Total generated tokens: 170000 Total generated tokens (retokenized): 169983 Request throughput (req/s): 3.73 Input token throughput (tok/s): 14858.12 Output token throughput (tok/s): 1982.80 Peak output token throughput (tok/s): 3734.00 Peak concurrent requests: 70 Total token throughput (tok/s): 16840.92 Concurrency: 59.75 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 16008.12 Median E2E Latency (ms): 15460.65 P90 E2E Latency (ms): 27705.81 P99 E2E Latency (ms): 32874.74 ---------------Time to First Token---------------- Mean TTFT (ms): 476.99 Median TTFT (ms): 177.50 P99 TTFT (ms): 3014.39 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 29.81 Median TPOT (ms): 31.19 P99 TPOT (ms): 45.53 ---------------Inter-Token Latency---------------- Mean ITL (ms): 29.29 Median ITL (ms): 15.75 P95 ITL (ms): 173.94 P99 ITL (ms): 202.00 Max ITL (ms): 2783.23 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python benchmark/gsm8k/bench_sglang.py --port 30000 ``` * **Test Results:** ```text Output theme={null} Accuracy: 0.965 Invalid: 0.000 Latency: 26.407 s Output throughput: 929.132 token/s ``` #### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} cd benchmark/mmlu bash download_data.sh python3 bench_sglang.py --port 30000 ``` * **Test Results:** ```text Output theme={null} subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.807 subject: astronomy, #q:152, acc: 0.921 subject: business_ethics, #q:100, acc: 0.820 subject: clinical_knowledge, #q:265, acc: 0.860 subject: college_biology, #q:144, acc: 0.944 subject: college_chemistry, #q:100, acc: 0.590 subject: college_computer_science, #q:100, acc: 0.820 subject: college_mathematics, #q:100, acc: 0.800 subject: college_medicine, #q:173, acc: 0.803 subject: college_physics, #q:102, acc: 0.775 subject: computer_security, #q:100, acc: 0.880 subject: conceptual_physics, #q:235, acc: 0.936 subject: econometrics, #q:114, acc: 0.807 subject: electrical_engineering, #q:145, acc: 0.834 subject: elementary_mathematics, #q:378, acc: 0.854 subject: formal_logic, #q:126, acc: 0.802 subject: global_facts, #q:100, acc: 0.610 subject: high_school_biology, #q:310, acc: 0.971 subject: high_school_chemistry, #q:203, acc: 0.803 subject: high_school_computer_science, #q:100, acc: 0.920 subject: high_school_european_history, #q:165, acc: 0.891 subject: high_school_geography, #q:198, acc: 0.929 subject: high_school_government_and_politics, #q:193, acc: 0.969 subject: high_school_macroeconomics, #q:390, acc: 0.903 subject: high_school_mathematics, #q:270, acc: 0.689 subject: high_school_microeconomics, #q:238, acc: 0.962 subject: high_school_physics, #q:151, acc: 0.854 subject: high_school_psychology, #q:545, acc: 0.947 subject: high_school_statistics, #q:216, acc: 0.815 subject: high_school_us_history, #q:204, acc: 0.907 subject: high_school_world_history, #q:237, acc: 0.937 subject: human_aging, #q:223, acc: 0.821 subject: human_sexuality, #q:131, acc: 0.840 subject: international_law, #q:121, acc: 0.934 subject: jurisprudence, #q:108, acc: 0.870 subject: logical_fallacies, #q:163, acc: 0.847 subject: machine_learning, #q:112, acc: 0.812 subject: management, #q:103, acc: 0.922 subject: marketing, #q:234, acc: 0.923 subject: medical_genetics, #q:100, acc: 0.970 subject: miscellaneous, #q:783, acc: 0.941 subject: moral_disputes, #q:346, acc: 0.850 subject: moral_scenarios, #q:895, acc: 0.726 subject: nutrition, #q:306, acc: 0.915 subject: philosophy, #q:311, acc: 0.859 subject: prehistory, #q:324, acc: 0.889 subject: professional_accounting, #q:282, acc: 0.723 subject: professional_law, #q:1534, acc: 0.648 subject: professional_medicine, #q:272, acc: 0.923 subject: professional_psychology, #q:612, acc: 0.845 subject: public_relations, #q:110, acc: 0.782 subject: security_studies, #q:245, acc: 0.796 subject: sociology, #q:201, acc: 0.925 subject: us_foreign_policy, #q:100, acc: 0.950 subject: virology, #q:166, acc: 0.572 subject: world_religions, #q:171, acc: 0.883 Total latency: 208.985 Average accuracy: 0.834 ``` # Qwen3-Next Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Next ## 1. Model Introduction [Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding. Qwen3-Next introduces several groundbreaking innovations: * **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts. * **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity. * **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed. * **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications. * **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use. * **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training. For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd\&from=research.latest-advancements-list). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload. * `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`. * `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`. * **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser 1. **Streaming with Thinking Process:** Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections. ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Thinking \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36. =============== Content ================= To find **15% of 240**, follow these steps: --- ### **Step 1: Understand what "percent" means** - "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form. --- ### **Step 2: Multiply the number by the decimal** - To find 15% of 240, multiply: $$ 240 \times 0.15 $$ --- ### **Step 3: Break it down for clarity (optional but helpful)** - **10% of 240** = $ 240 \times 0.1 = 24 $ - **5% of 240** = $ 240 \times 0.05 = 12 $ - Add them together: $$ 24 + 12 = 36 $$ --- ### **Step 4: Confirm with direct multiplication** - $ 240 \times 0.15 = 36 $ --- ### ✅ Final Answer: $$ \boxed{36} $$ ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. 2. **Turn off Thinking:** Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode. ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Turn off thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Instruct", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True, extra_body={"chat_template_kwargs": {"enable_thinking": False}} ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find **15% of 240**, follow these steps: --- ### **Step 1: Understand what percentage means** "Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction: $$ \frac{15}{100} $$ --- ### **Step 2: Multiply the fraction by the number** To find 15% of 240, multiply: $$ \frac{15}{100} \times 240 $$ --- ### **Step 3: Simplify the multiplication** You can simplify this in a couple of ways. #### **Option A: Multiply first, then divide** $$ 15 \times 240 = 3600 $$ Then divide by 100: $$ \frac{3600}{100} = 36 $$ #### **Option B: Simplify the fraction first** $$ \frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)} $$ Now multiply: $$ \frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36 $$ --- ### **Step 4: Final Answer** $$ \boxed{36} $$ So, **15% of 240 is 36**. ``` #### 4.2.2 Tool Calling Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser: **Python Example (without Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Instruct", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} {"name": "get_weather", "arguments": {"location": "Beijing"}} ``` **Python Example (with Thinking Process):** Start sglang server: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Thinking \ --reasoning-parser qwen3 \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 8000 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"🔧 Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}. {"name": "get_weather", "arguments": {"location": "Beijing"}} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-Next-80B-A3B-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` #### 4.2.3 Processing Ultra-Long Texts Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method. **Qwen3-Next-80B-A3B-Instruct** ```shell Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000 ``` **Qwen3-Next-80B-A3B-Thinking** ```shell Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking --reasoning-parser qwen3 --tp 8 --host 0.0.0.0 --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000 ``` #### 4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding) Qwen3-Next ships built-in Multi-Token Prediction (MTP) layers and supports [EAGLE-style speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding) through the `NEXTN` algorithm. The MTP weights are bundled in the main checkpoint, so no separate draft model is required. ```shell Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` Tune `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens` for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). See [PR #10233](https://github.com/sgl-project/sglang/pull/10233) for implementation details. ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Tensor Parallelism: 8 * Model: Qwen/Qwen3-Next-80B-A3B-Instruct * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Latency-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 100 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 146.52 Total input tokens: 33839 Total input text tokens: 33839 Total input vision tokens: 0 Total generated tokens: 21640 Total generated tokens (retokenized): 21619 Request throughput (req/s): 0.68 Input token throughput (tok/s): 230.95 Output token throughput (tok/s): 147.70 Peak output token throughput (tok/s): 164.00 Peak concurrent requests: 6 Total token throughput (tok/s): 378.65 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1464.81 Median E2E Latency (ms): 1077.48 ---------------Time to First Token---------------- Mean TTFT (ms): 127.88 Median TTFT (ms): 132.88 P99 TTFT (ms): 212.85 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 6.19 Median TPOT (ms): 6.17 P99 TPOT (ms): 6.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.21 Median ITL (ms): 6.16 P95 ITL (ms): 6.51 P99 ITL (ms): 6.71 Max ITL (ms): 10.07 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Server Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-Next-80B-A3B-Instruct \ --tp 8 \ ``` * Test Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --num-prompt 1000 \ --max-concurrency 100 ``` **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 100.32 Total input tokens: 302118 Total input text tokens: 302118 Total input vision tokens: 0 Total generated tokens: 195775 Total generated tokens (retokenized): 195016 Request throughput (req/s): 9.97 Input token throughput (tok/s): 3011.69 Output token throughput (tok/s): 1951.60 Peak output token throughput (tok/s): 5909.00 Peak concurrent requests: 120 Total token throughput (tok/s): 4963.29 Concurrency: 93.05 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9333.98 Median E2E Latency (ms): 6054.12 ---------------Time to First Token---------------- Mean TTFT (ms): 161.77 Median TTFT (ms): 137.94 P99 TTFT (ms): 503.29 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 50.87 Median TPOT (ms): 50.28 P99 TPOT (ms): 122.87 ---------------Inter-Token Latency---------------- Mean ITL (ms): 47.11 Median ITL (ms): 13.84 P95 ITL (ms): 195.33 P99 ITL (ms): 289.56 Max ITL (ms): 486.38 ================================================== ``` ### 5.2 Accuracy Benchmark ### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 --port 8000 ``` * **Results**: * Qwen3-Next-80B-A3B-Instruct ``` Accuracy: 0.960 Invalid: 0.000 Latency: 12.673 s Output throughput: 2538.255 token/s ``` * Qwen3-Next-80B-A3B-Thinking ``` Accuracy: 0.935 Invalid: 0.000 Latency: 9.912 s Output throughput: 3288.737 token/s ``` ### 5.2.2 MMLU Benchmark * **Benchmark Command:** ```shell Command theme={null} cd sglang bash benchmark/mmlu/download_data.sh python3 benchmark/mmlu/bench_sglang.py --nsub 10 ``` * **Results**: * Qwen3-Next-80B-A3B-Instruct ``` subject: abstract_algebra, #q:100, acc: 0.800 subject: anatomy, #q:135, acc: 0.807 subject: astronomy, #q:152, acc: 0.947 subject: business_ethics, #q:100, acc: 0.810 subject: clinical_knowledge, #q:265, acc: 0.894 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.680 subject: college_computer_science, #q:100, acc: 0.860 subject: college_mathematics, #q:100, acc: 0.780 subject: college_medicine, #q:173, acc: 0.861 Total latency: 10.098 Average accuracy: 0.856 ``` * Qwen3-Next-80B-A3B-Thinking ``` subject: abstract_algebra, #q:100, acc: 0.780 subject: anatomy, #q:135, acc: 0.815 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.870 subject: clinical_knowledge, #q:265, acc: 0.894 subject: college_biology, #q:144, acc: 0.965 subject: college_chemistry, #q:100, acc: 0.670 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.770 subject: college_medicine, #q:173, acc: 0.861 Total latency: 10.236 Average accuracy: 0.855 ``` # Qwen3-VL Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-VL ## 1. Model Introduction [Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications. This generation delivers comprehensive upgrades across the board: * **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942) * **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities. * **Extended context length**: Supports up to 262K tokens for processing long documents and videos. * **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics. * **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance. * **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions. For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility. * **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.) * **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K. * **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details. * **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton). **Hardware-specific notes:** * **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency. * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage. * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. **Additional multimodal server parameters:** * `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid device-to-host memory copies, improving performance for high-frequency inference. **Example with full multimodal optimizations:** ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 \ SGLANG_VLM_CACHE_SIZE_MB=0 \ python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-235B-A22B-Instruct \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code \ --tp-size 8 \ --enable-cache-report \ --log-level info \ --max-running-requests 64 \ --mem-fraction-static 0.65 \ --chunked-prefill-size 8192 \ --attention-backend fa3 \ --mm-attention-backend fa3 \ --enable-metrics ``` * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Qwen3-VL supports both image and video inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.37s Generated text: Auntie Anne's CINNAMON SUGAR 1 x 17,000 17,000 SUB TOTAL 17,000 GRAND TOTAL 17,000 CASH IDR 20,000 CHANGE DUE 3,000 ``` **Multi-Image Input Example:** Qwen3-VL can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 10.18s Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering. The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy. In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout. Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance. In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose. ``` **Video Input Example:** Qwen3-VL supports video understanding by processing video URLs: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ] start = time.time() response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Instruct", messages=messages, max_tokens=2048 ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Note:** * For video processing, ensure you have sufficient context length configured (up to 262K tokens) * Video processing may require more memory; adjust `--mem-fraction-static` accordingly * You can also provide local file paths using `file://` protocol **Example Output:** ```text Output theme={null} Response costs: 3.89s Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen. ``` #### 4.2.2 Reasoning Parser Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Thinking \ --reasoning-parser qwen3 \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= To solve this problem, I need to calculate 15% of 240. Step 1: Convert 15% to decimal: 15% = 0.15 Step 2: Multiply 240 by 0.15 Step 3: 240 × 0.15 = 36 =============== Content ================= The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36. ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Qwen3-VL supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Thinking \ --reasoning-parser qwen3 \ --tool-call-parser qwen \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"🔧 Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. I should call the function with location="Beijing". =============== Content ================= 🔧 Tool Call: get_weather Arguments: {"location": "Beijing", "unit": "celsius"} ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="Qwen/Qwen3-VL-235B-A22B-Thinking", messages=messages, temperature=0.7 ) print(final_response.choices[0].message.content) # Output: "The weather in Beijing is currently 22°C and sunny." ``` ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (8x) * Model: Qwen3-VL-235B-A22B-Instruct * Tensor Parallelism: 8 * sglang version: 0.5.6 We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 45.97 Total input tokens: 18348 Total input text tokens: 708 Total input vision tokens: 17640 Total generated tokens: 4220 Total generated tokens (retokenized): 3423 Request throughput (req/s): 0.22 Input token throughput (tok/s): 399.17 Output token throughput (tok/s): 91.81 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 2 Total token throughput (tok/s): 490.98 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 4594.52 Median E2E Latency (ms): 3725.04 ---------------Time to First Token---------------- Mean TTFT (ms): 193.35 Median TTFT (ms): 196.32 P99 TTFT (ms): 222.75 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.44 Median TPOT (ms): 10.44 P99 TPOT (ms): 10.47 ---------------Inter-Token Latency---------------- Mean ITL (ms): 11.78 Median ITL (ms): 10.48 P95 ITL (ms): 21.01 P99 ITL (ms): 31.40 Max ITL (ms): 31.92 ================================================== ``` **Optimized Results (with CUDA IPC Transport):** For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features. * Model Deployment Command: ```shell Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 100 \ --max-concurrency 1 ``` * **Test Results:** With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 100 Benchmark duration (s): 566.84 Total input tokens: 183667 Total input text tokens: 7267 Total input vision tokens: 176400 Total generated tokens: 52444 Total generated tokens (retokenized): 28702 Request throughput (req/s): 0.18 Input token throughput (tok/s): 324.02 Output token throughput (tok/s): 92.52 Peak output token throughput (tok/s): 96.00 Peak concurrent requests: 3 Total token throughput (tok/s): 416.54 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 5667.50 Median E2E Latency (ms): 5830.00 ---------------Time to First Token---------------- Mean TTFT (ms): 191.16 Median TTFT (ms): 182.58 P99 TTFT (ms): 244.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 10.46 Median TPOT (ms): 10.46 P99 TPOT (ms): 10.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 13.91 Median ITL (ms): 10.56 P95 ITL (ms): 21.35 P99 ITL (ms): 31.55 Max ITL (ms): 42.36 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --tp 8 \ --host 0.0.0.0 \ --port 30000 ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3-VL-235B-A22B-Instruct \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 584.65 Total input tokens: 1839015 Total input text tokens: 75015 Total input vision tokens: 1764000 Total generated tokens: 510855 Total generated tokens (retokenized): 284284 Request throughput (req/s): 1.71 Input token throughput (tok/s): 3145.50 Output token throughput (tok/s): 873.78 Peak output token throughput (tok/s): 2855.00 Peak concurrent requests: 107 Total token throughput (tok/s): 4019.29 Concurrency: 98.35 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 57502.05 Median E2E Latency (ms): 54301.08 ---------------Time to First Token---------------- Mean TTFT (ms): 5802.23 Median TTFT (ms): 1444.75 P99 TTFT (ms): 46675.92 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 100.22 Median TPOT (ms): 105.43 P99 TPOT (ms): 144.37 ---------------Inter-Token Latency---------------- Mean ITL (ms): 134.20 Median ITL (ms): 25.57 P95 ITL (ms): 558.14 P99 ITL (ms): 1449.01 Max ITL (ms): 33453.23 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`: * Benchmark Command: ```shell Command theme={null} uv pip install lmms_eval python3 -m lmms_eval \ --model openai_compatible \ --model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \ --tasks mmmu_val \ --batch_size 128 \ --log_samples \ --log_samples_suffix "openai_compatible" \ --output_path ./logs \ --gen_kwargs "max_new_tokens=4096" ``` * **Test Results:** ```text Output theme={null}
Tasks Version Filter n-shot Metric Value Stderr
mmmu_val 0 none 0 mmmu_acc ↑ 0.6567 ± N/A
``` # Qwen3.5 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5 ## 1. Model Introduction [Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is the latest flagship model in the Qwen series developed by Alibaba, representing a significant leap forward with unified vision-language foundation, efficient hybrid architecture, and scalable reinforcement learning. Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated), delivering high-throughput inference with minimal latency. It supports multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens. **Architecture details:** * **Hybrid Attention:** Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer — linear layers provide low-cost long-context processing while periodic full attention ensures high associative recall. * **MoE routing:** Top-10 active out of 512 routed experts plus a dedicated shared expert for universal features, keeping 17B parameters active from 397B total. * **Native multimodal:** DeepStack Vision Transformer with Conv3d temporal encoding for image and video understanding without separate visual encoders. **Key Features:** * **Unified Vision-Language Foundation**: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models * **Efficient Hybrid Architecture**: Gated Delta Networks + sparse MoE (397B total / 17B active) for high-throughput inference * **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses * **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser * **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency * **201 Language Support**: Expanded multilingual coverage across 201 languages and dialects **Available Models:**
Model BF16 (Full precision) FP8 (8-bit Quantized) FP4 (4-bit Quantized)
Qwen3.5-397B-A17B [Qwen/Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) NVIDIA NVFP4: [nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)
AMD MXFP4: [amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4)
Qwen3.5-122B-A10B [Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) [Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8) -
Qwen3.5-35B-A3B [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) [Qwen/Qwen3.5-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8) -
Qwen3.5-27B [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) [Qwen/Qwen3.5-27B-FP8](https://huggingface.co/Qwen/Qwen3.5-27B-FP8) -
Qwen3.5-9B [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) - -
Qwen3.5-4B [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) - -
Qwen3.5-2B [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) - -
Qwen3.5-0.8B [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) - -
**License:** Apache 2.0 ## 2. SGLang Installation SGLang from the main branch is required for Qwen3.5. You can install from source or use a Docker image: ```bash Command theme={null} # Install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker (NVIDIA GPUs) docker pull lmsysorg/sglang:latest # Or use Docker (AMD MI300X/MI325X) docker pull lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi30x-20260715 # Or use Docker (AMD MI355X) docker pull lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260715 ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. * **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). For the **MXFP4 checkpoint on MI355X**, set `ROCM_QUICK_REDUCE_QUANTIZATION=INT8` to route multi-GPU collectives through INT8-quantized ROCm quick all-reduce, and drop `--enable-aiter-allreduce-fusion` (the two are mutually exclusive; quick all-reduce is preferred for this recipe). * **KV cache offloading (MXFP4 on MI355X):** Selecting **Host DRAM (HiCache)** adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend direct --hicache-mem-layout page_first_direct`. HiCache extends the radix cache, so the generated command drops `--disable-radix-cache` — the two options are mutually exclusive and SGLang rejects them together at startup. `--hicache-ratio 1.5` sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this **Disabled** for fixed-length throughput serving, where the extra host traffic buys nothing. * **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. * **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * **B200 (FP8)**: Add `--enable-flashinfer-allreduce-fusion` for optimized throughput on Blackwell. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Hardware requirements: * **BF16**: \~397B parameters require \~800GB of GPU memory for weights. * **H100 (80GB)** requires tp=16 (2 nodes) since each rank needs \~100GB at tp=8. * **H200 (141GB)** runs with tp=8. * **B200 (183GB)** runs with tp=8. * **B300 (275GB)** runs with tp=4. * **MI300X (192GB)** runs with tp=8. * **MI325X (256GB)** runs with tp=4. * **MI355X (288GB)** runs with tp=4. * **FP8**: The FP8 quantized model requires \~400GB for weights, cutting memory in half. * **H100 (80GB)** runs with tp=8. * **H200 (141GB)** runs with tp=4. * **B200 (183GB)** runs with tp=4. * **B300 (275GB)** runs with tp=2. * **MI300X (192GB)** runs with tp=4. * **MI325X (256GB)** runs with tp=2. * **MI355X (288GB)** runs with tp=2. * **FP4**: The FP4 quantized model requires \~250GB for weights, cutting memory by almost 4x. NVFP4 ([nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint ([amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4)) for MI355X. * **B200 (183GB)** runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4) * **B300 (275GB)** runs with tp=2. (NVFP4) * **MI355X (288GB)** runs with tp=2 (use tp=4 for low concurrency). (MXFP4)
Hardware Memory BF16 TP FP8 TP FP4 TP
H100 80GB 16 8 N/A
H200 141GB 8 4 N/A
B200 183GB 8 4 4 / 2 + EP2 (MTP)
B300 275GB 4 2 2
MI300X 192GB 8 4 N/A
MI325X 256GB 4 2 N/A
MI355X 288GB 4 2 2 / 4 (MXFP4)
**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is emitted only for the MXFP4-on-MI355X recipe, where it is part of the tuned configuration; for every other combination the generated commands leave it out, so add it manually only if memory constraints require the trade-off. * **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation **NVIDIA:** Deploy Qwen3.5-397B-A17B with the following command (H200, all features enabled): ```shell Command theme={null} sglang serve \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algo NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` **AMD:** Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X): ```shell Command theme={null} SGLANG_USE_AITER=1 \ SGLANG_USE_AITER_UNIFIED_ATTN=1 \ AITER_FLYDSL_FORCE=1 \ SGLANG_MAMBA_SSM_DTYPE=bfloat16 \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --trust-remote-code \ --page-size 16 \ --mem-fraction-static 0.8 \ --attention-backend aiter \ --enable-aiter-allreduce-fusion \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --watchdog-timeout 1200 \ --host 0.0.0.0 \ --port 30000 ``` > **Note:** To enable MTP (speculative decoding), append `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Vision Input Qwen3.5 supports image and video inputs as a unified vision-language model. Here is an example with an image: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user wants a detailed description of the provided image. 1. **Identify the main components:** * There is a central square. * There are four circles of varying sizes. 2. **Analyze the arrangement and relationships:** * **The Square:** It's in the middle, oriented upright (sides are vertical and horizontal). It's drawn with a thin black line. * **The Circles:** * **Top Right:** A large circle. It overlaps the top right corner of the square. A significant portion of the circle is outside the square, but a chunk of it is inside. * **Bottom Left:** A large circle, roughly the same size as the top right one. It overlaps the bottom left corner of the square. Again, a chunk is inside, most is outside. * **Top Left:** A smaller circle. It is positioned near the top left corner of the square. It overlaps the corner slightly. * **Bottom Right:** A smaller circle, roughly the same size as the top left one. It is positioned near the bottom right corner of the square. It overlaps the corner slightly. 3. **Synthesize the description:** * Start with a general overview: A geometric line drawing. * Describe the central shape: A square. * Describe the surrounding shapes: Four circles. * Detail the positioning: * Two large circles are diagonally opposite (bottom-left and top-right). * Two smaller circles are diagonally opposite (top-left and bottom-right). * Describe the interaction/overlap: The shapes are not just touching; they intersect. The lines cross over each other. The square acts as a central anchor. The circles seem to be "orbiting" or clustered around the corners of the square. * Mention the style: Black lines on a white background. Minimalist. No shading or color. 4. **Refine the details:** * Let's look closer at the overlaps. * **Top Left Circle:** It sits mostly above and to the left of the square's top-left corner. The bottom-right part of this small circle cuts into the square. * **Top Right Circle:** This is the largest circle (or tied for largest). Its bottom-left quadrant overlaps significantly with the top-right quadrant of the square. * **Bottom Left Circle:** This is also a large circle. Its top-right quadrant overlaps significantly with the bottom-left quadrant of the square. * **Bottom Right Circle:** This is a small circle. Its top-left part overlaps with the bottom-right corner of the square. 5. **Final Polish:** Structure the description logically. Start with the big picture, then break down each element's position relative to the center. * *Drafting the description:* The image shows a black and white line drawing of geometric shapes. In the center is a square. Surrounding it are four circles of two different sizes. * *Adding detail:* There are two large circles and two small circles. * *Positioning:* * A large circle is in the bottom left. It overlaps the bottom-left corner of the square. * A large circle is in the top right. It overlaps the top-right corner of the square. * A small circle is in the top left. It overlaps the top-left corner of the square. * A small circle is in the bottom right. It overlaps the bottom-right corner of the square. * *Visual style:* The lines are thin and black. The background is white. The shapes are transparent (wireframe style), so you can see the lines of the square through the circles and vice versa where they overlap. 6. **Final Review against the image:** Does this match? Yes. The description is accurate. The "transparency" point is important because it's a line drawing, not solid shapes. The lines cross. * Wait, let me look really closely at the "Top Left" circle. It actually looks like it's touching the top edge and left edge. It overlaps the corner. * Let me look at the "Bottom Right" circle. It overlaps the bottom right corner. * The arrangement creates a sort of diagonal symmetry. Large-Large on one diagonal, Small-Small on the other. Let's write the final output based on these observations. =============== Content ================= Based on the image provided, here is a detailed description: **Overview** The image is a minimalist, black-and-white line drawing featuring geometric shapes. It consists of a central square surrounded by four circles of varying sizes. The lines are thin and black against a plain white background. The shapes are drawn in a "wireframe" style, meaning they are transparent outlines; where shapes overlap, the lines cross over each other rather than one blocking the other. **Detailed Breakdown** 1. **The Central Square:** * There is a single square positioned in the center of the composition. It is oriented upright with vertical and horizontal sides. 2. **The Circles:** * There are four circles arranged around the corners of the square. They appear in two distinct sizes: two large circles and two smaller circles. * **Top Right:** A large circle is positioned at the top right. It overlaps significantly with the top-right corner of the square. A portion of the circle's interior is inside the square's boundary. * **Bottom Left:** Another large circle (roughly the same size as the top right one) is positioned at the bottom left. It overlaps significantly with the bottom-left corner of the square. * **Top Left:** A smaller circle is positioned near the top left corner. It overlaps slightly with the top-left corner of the square. * **Bottom Right:** A smaller circle (roughly the same size as the top left one) is positioned near the bottom right corner. It overlaps slightly with the bottom-right corner of the square. **Composition** The arrangement creates a diagonal symmetry. The two largest circles are on a diagonal from bottom-left to top-right, while the two smallest circles are on a diagonal from top-left to bottom-right. The intersecting lines create a complex web of curves and angles in the center of the image. ``` ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser Qwen3.5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Thinking mode is enabled by default, no extra parameters needed response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= Thinking Process: 1. **Analyze the Request:** The user wants to solve a math problem: "What is 15% of 240?" and expects a step-by-step explanation. 2. **Identify the Core Operation:** The problem asks for a percentage of a number. The formula is: $\text{Percentage} \times \text{Number} = \text{Result}$. * Percentage: $15\%$ * Number: $240$ 3. **Determine the Steps:** * Step 1: Convert the percentage to a decimal or fraction. * Step 2: Multiply the converted value by the number. * Step 3: Calculate the final result. * Alternative Step (Mental Math): Break down 15% into 10% + 5%. 4. **Draft the Explanation (Method 1: Decimal Conversion):** * Convert $15\%$ to $0.15$. * Multiply $0.15 \times 240$. * Calculation: $15 \times 24 = 360$, adjust decimal places $\rightarrow 36.0$. 5. **Draft the Explanation (Method 2: Fraction Conversion):** * Convert $15\%$ to $\frac{15}{100}$ or $\frac{3}{20}$. * Multiply $\frac{3}{20} \times 240$. * $240 / 20 = 12$. * $3 \times 12 = 36$. 6. **Draft the Explanation (Method 3: Breaking it down - often easier for step-by-step):** * Find 10% of 240. * Find 5% of 240 (which is half of 10%). * Add them together. * 10% of 240 = 24. * 5% of 240 = 12. * $24 + 12 = 36$. 7. **Select the Best Presentation:** Method 3 is very intuitive for "step-by-step" requests as it shows logical reasoning without heavy multiplication. However, the standard decimal multiplication (Method 1) is the most formal. I will present the standard method clearly, perhaps mentioning the breakdown method as a check or alternative way to understand it. Let's stick to the standard multiplication method for clarity but explain the conversion clearly. Actually, providing the breakdown method is often more helpful for understanding *why*. I'll provide the standard calculation method as the primary steps. * Step 1: Convert percentage to decimal. * Step 2: Multiply. * Step 3: State answer. Let's refine the calculation for the final output. $15\% = 0.15$ $0.15 \times 240$ $= \frac{15}{100} \times 240$ $= 15 \times 2.4$ $= 36$ Or: $10\% = 24$ $5\% = 12$ $15\% = 36$ I will write out the solution using the decimal/fraction multiplication method as it's universally applicable, but I'll make the steps very clear. 8. **Final Review:** Does this answer the prompt accurately and step-by-step? Yes. * Step 1: Convert 15% to a decimal. * Step 2: Multiply by 240. * Step 3: Calculate. Let's write it out. Step 1: Convert the percentage to a decimal. $15\% = 0.15$ Step 2: Multiply the decimal by the number. $0.15 \times 240$ Step 3: Perform the multiplication. $15 \times 240 = 3600$ Adjust decimal (2 places): $36.00$ Result: 36. Wait, $0.15 \times 240$. $15 \times 24 = 360$. One zero from 240 cancels one decimal place? $0.15 \times 240 = 15 \times 2.4 = 36$. Or $\frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36$. This looks correct. I will also include the "10% + 5%" method as an alternative way to think about it, as it's often easier for mental math. Plan: 1. Define the formula. 2. Convert percent to decimal/fraction. 3. Multiply. 4. Provide the alternative mental math method for completeness. 5. State final answer clearly. Actually, keeping it simple is better. Just one clear path. I'll use the fraction/decimal multiplication path. Step 1: Convert 15% to a decimal. Step 2: Multiply 240 by 0.15. Step 3: Final result. Let's go.cw =============== Content ================= Here is the step-by-step solution to find 15% of 240: **Step 1: Convert the percentage to a decimal.** To convert a percentage to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ **Step 2: Multiply the decimal by the number.** Now, multiply 0.15 by 240. $$0.15 \times 240$$ **Step 3: Calculate the result.** You can think of this as: $$15 \times 240 = 3600$$ Since there are two decimal places in 0.15, move the decimal point in the result two places to the left: $$3600 \rightarrow 36.00$$ **Alternative Method (Mental Math):** * Find 10% of 240: $240 \div 10 = 24$ * Find 5% of 240 (half of 10%): $24 \div 2 = 12$ * Add them together (10% + 5% = 15%): $24 + 12 = 36$ **Answer:** 15% of 240 is **36**. ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Disable thinking mode via chat_template_kwargs response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) # In Instruct mode, the model responds directly without reasoning_content for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} To find 15% of 240, you can follow these steps: ### Step-by-Step Deduction 1. **Convert the percentage to a decimal **: To convert a percentage to a decimal, divide by 100. $$15\% = \frac{15}{100} = 0.15$$ 2. **Multiply the decimal by the number**: Multiply $0.15$ by $240$. $$0.15 \times 240$$ *Alternative Method (Mental Math)*: - Find 10% of 240: $240 \times 0.10 = 24$ - Find 5% of 240 (which is half of 10%): $24 / 2 = 12$ - Add them together ($10\% + 5\% = 15\%$): $24 + 12 = 36$ 3. **Calculation**: $$240 \times 0.15 = 36$$ ### Final Conclusion 15% of 240 is **36**. ``` #### 4.3.2 Tool Calling Qwen3.5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`. **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="Qwen/Qwen3.5-397B-A17B", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) # Process streaming response thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") # Print content if delta.content: print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I have access to a get_weather function that can provide current weather information for a location. Let me check the parameters: - location (required): "Beijing" - this is provided by the user - unit (optional): The user didn't specify a temperature unit, so I won't include this optional parameter I should call the get_weather function with Beijing as the location. =============== Content ================= Tool Call: get_weather Arguments: Tool Call: None Arguments: { Tool Call: None Arguments: "location": "Beijing" Tool Call: None Arguments: } ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --port 30000 ``` * Test Result ```text Output theme={null} 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:31<00:00, 6.43it/s] Accuracy: 0.975 Invalid: 0.005 Latency: 31.784 s Output throughput: 998.166 token/s ``` #### 5.1.2 GSM8K with lm-eval (5-shot) Evaluate using the industry-standard `lm-eval` harness for reproducible accuracy reporting: ```bash Command theme={null} pip install lm-eval[api] lm_eval --model local-completions \ --model_args '{"base_url": "http://localhost:30000/v1/completions", "model": "Qwen/Qwen3.5-397B-A17B", "num_concurrent": 256, "max_retries": 10, "max_gen_toks": 2048}' \ --tasks gsm8k \ --batch_size auto \ --num_fewshot 5 \ --trust_remote_code ``` #### 5.1.3 MMMU Benchmark * Benchmark Command ```bash Command theme={null} python3 benchmark/mmmu/bench_sglang.py --concurrency 128 --port 30000 --max-new-tokens 512 ``` * Test Result ```text Output theme={null} {'Accounting': {'acc': 1.0, 'num': 3}, 'Agriculture': {'acc': 1.0, 'num': 4}, 'Art': {'acc': 1.0, 'num': 9}, 'Art_Theory': {'acc': 1.0, 'num': 5}, 'Basic_Medical_Science': {'acc': 1.0, 'num': 2}, 'Biology': {'acc': 1.0, 'num': 1}, 'Chemistry': {'acc': 1.0, 'num': 1}, 'Computer_Science': {'acc': 1.0, 'num': 1}, 'Design': {'acc': 0.909, 'num': 11}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 1.0, 'num': 1}, 'Economics': {'acc': 1.0, 'num': 5}, 'Finance': {'acc': 1.0, 'num': 2}, 'Geography': {'acc': 1.0, 'num': 3}, 'History': {'acc': 1.0, 'num': 3}, 'Literature': {'acc': 0.938, 'num': 16}, 'Manage': {'acc': 1.0, 'num': 2}, 'Marketing': {'acc': 1.0, 'num': 5}, 'Math': {'acc': 1.0, 'num': 1}, 'Overall': {'acc': 0.978, 'num': 91}, 'Overall-Art and Design': {'acc': 0.96, 'num': 25}, 'Overall-Business': {'acc': 1.0, 'num': 17}, 'Overall-Health and Medicine': {'acc': 1.0, 'num': 7}, 'Overall-Humanities and Social Science': {'acc': 0.966, 'num': 29}, 'Overall-Science': {'acc': 1.0, 'num': 8}, 'Overall-Tech and Engineering': {'acc': 1.0, 'num': 5}, 'Pharmacy': {'acc': 1.0, 'num': 2}, 'Physics': {'acc': 1.0, 'num': 2}, 'Psychology': {'acc': 1.0, 'num': 4}, 'Public_Health': {'acc': 1.0, 'num': 2}, 'Sociology': {'acc': 1.0, 'num': 6}} eval out saved to ./val_sglang.json Overall accuracy: 0.978 ``` ### 5.2 Speed Benchmark **Test Environment:** * Hardware: H200 (8x) * Model: Qwen3.5-397B-A17B * Tensor Parallelism: 8 * SGLang Version: main branch Server Launch Command: ```bash Command theme={null} SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \ --model Qwen/Qwen3.5-397B-A17B \ --tp 8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algo NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` #### 5.2.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 18.94 Total input tokens: 6101 Total input text tokens: 6101 Total generated tokens: 4220 Total generated tokens (retokenized): 4211 Request throughput (req/s): 0.53 Input token throughput (tok/s): 322.16 Output token throughput (tok/s): 222.84 Peak output token throughput (tok/s): 289.00 Peak concurrent requests: 3 Total token throughput (tok/s): 545.00 Concurrency: 1.00 Accept length: 3.12 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1892.35 Median E2E Latency (ms): 1410.85 P90 E2E Latency (ms): 3749.34 P99 E2E Latency (ms): 4216.52 ---------------Time to First Token---------------- Mean TTFT (ms): 190.40 Median TTFT (ms): 208.46 P99 TTFT (ms): 261.27 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 3.96 Median TPOT (ms): 3.79 P99 TPOT (ms): 4.96 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.04 Median ITL (ms): 3.15 P95 ITL (ms): 6.65 P99 ITL (ms): 12.60 Max ITL (ms): 58.03 ================================================== ``` #### 5.2.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 283.04 Total input tokens: 502493 Total input text tokens: 502493 Total generated tokens: 500251 Total generated tokens (retokenized): 498222 Request throughput (req/s): 3.53 Input token throughput (tok/s): 1775.37 Output token throughput (tok/s): 1767.45 Peak output token throughput (tok/s): 3630.00 Peak concurrent requests: 108 Total token throughput (tok/s): 3542.82 Concurrency: 96.71 Accept length: 3.31 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 27372.05 Median E2E Latency (ms): 26660.21 P90 E2E Latency (ms): 39951.91 P99 E2E Latency (ms): 48405.51 ---------------Time to First Token---------------- Mean TTFT (ms): 14247.21 Median TTFT (ms): 14932.44 P99 TTFT (ms): 20998.45 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.16 Median TPOT (ms): 26.13 P99 TPOT (ms): 41.33 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.29 Median ITL (ms): 11.38 P95 ITL (ms): 72.10 P99 ITL (ms): 149.57 Max ITL (ms): 1220.68 ================================================== ``` #### 5.2.3 Agentic Long-Context with HiCache DRAM Offload (H200 FP8, MTP) For agentic workloads, here is how to enable HiCache and MTP. Container image (pinned for reproducibility): `lmsysorg/sglang:nightly-dev-cu13-20260815-a5ba081f`. Server Launch Command: ```bash Command theme={null} SGLANG_ENABLE_SPEC_V2=1 \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B-FP8 \ --served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \ --trust-remote-code \ --tensor-parallel-size 8 \ --data-parallel-size 1 \ --expert-parallel-size 1 \ --quantization fp8 \ --kv-cache-dtype fp8_e4m3 \ --mamba-ssm-dtype bfloat16 \ --attention-backend flashinfer \ --enable-flashinfer-allreduce-fusion \ --mem-fraction-static 0.8 \ --stream-interval 50 \ --scheduler-recv-interval 10 \ --tokenizer-worker-num 6 \ --enable-metrics \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --page-size 64 \ --enable-hierarchical-cache \ --hicache-size 77 \ --hicache-io-backend kernel \ --hicache-mem-layout page_first \ --hicache-write-policy write_through_selective ``` ### 5.3 Vision Speed Benchmark We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens. #### 5.3.1 Latency Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 \ --request-rate inf ``` ```text Output theme={null} TODO ``` #### 5.3.2 Throughput Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model Qwen/Qwen3.5-397B-A17B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 \ --request-rate inf ``` ```text Output theme={null} TODO ``` # Qwen3.6 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.6 ## 1. Model Introduction The Qwen3.6 series is developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation. Two size/sparsity variants are released: * [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — **Sparse MoE** (35B total, 3B active) on a Gated Delta Networks backbone. * [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — **Dense** hybrid GDN; smaller weights footprint, single-GPU friendly. Both variants share the same hybrid reasoning, tool-calling, and multimodal interface and natively handle context lengths of up to 262,144 tokens, extensible to over 1M tokens. **Key Features:** * **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision * **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development * **Efficient Hybrid Architecture**: Gated Delta Networks backbone; sparse MoE (35B / 3B active) or dense 27B variant * **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses * **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser * **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency; both MoE and Dense variants ship `mtp.safetensors` * **Multimodal**: Unified vision-language model supporting text, image, and video inputs **Available Models:**
Model Architecture Weights
Qwen3.6-35B-A3B (BF16) MoE 35B / 3B active [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
Qwen3.6-35B-A3B (FP8) MoE 35B / 3B active [Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)
Qwen3.6-35B-A3B (NVFP4) MoE 35B / 3B active (Blackwell) [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)
Qwen3.6-27B (BF16) Dense 27B [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)
Qwen3.6-27B (FP8) Dense 27B [Qwen/Qwen3.6-27B-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)
Qwen3.6-27B (NVFP4) Dense 27B (Blackwell) [nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
**License:** Apache 2.0 ## 2. SGLang Installation SGLang `>=0.5.10` is required for Qwen3.6. You can install from PyPI, from source, or use a Docker image: ```bash Command theme={null} # Install from PyPI uv pip install --prerelease=allow sglang # Or install from source uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' # Or use Docker (NVIDIA GPUs; also serves the NVFP4 variants) docker pull lmsysorg/sglang:latest ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities. ### 3.2 Configuration Tips * Speculative decoding (MTP) can significantly reduce latency for interactive use cases. * **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. * The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. * Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. * **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. * **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. * For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. * Hardware requirements: * **35B-A3B BF16**: \~70GB for weights. TP=1 fits on all supported hardware. * **35B-A3B FP8**: \~35GB for weights. TP=1 fits on all supported hardware. * **35B-A3B NVFP4**: \~23GB for weights. TP=1 fits on B200/B300. * **27B BF16**: \~54GB for weights. TP=1 fits on all supported hardware. * **27B FP8**: \~27GB for weights. TP=1 fits on all supported hardware. * **27B NVFP4**: \~22GB for weights. TP=1 fits on B200/B300. All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU. NVFP4 is available on B200/B300:
Hardware Memory BF16 TP FP8 TP NVFP4 TP
H100 80GB 1 1 —
H200 141GB 1 1 —
B200 183GB 1 1 1
B300 275GB 1 1 1
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over: ```shell Command theme={null} sglang serve \ --model-path Qwen/Qwen3.6-35B-A3B-FP8 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mem-fraction-static 0.8 \ --host 0.0.0.0 \ --port 30000 ``` ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Vision Input Qwen3.6 supports image and video inputs as a unified vision-language model. **Image Input Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg" } }, { "type": "text", "text": "Describe this image in detail." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Video Input Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": { "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4" } }, { "type": "text", "text": "Describe what happens in this video." } ] } ], max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` ### 4.3 Advanced Usage #### 4.3.1 Reasoning Parser Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response. To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time: * **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed. * **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process. **Example 1: Thinking Mode (Default)** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], max_tokens=2048, stream=True ) has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Example 2: Instruct Mode (Thinking Off)** To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "What is 15% of 240?"} ], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, max_tokens=2048, stream=True ) for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) print() ``` #### 4.3.2 Thinking Preservation Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency: ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "Help me plan a web app architecture."} ], extra_body={"chat_template_kwargs": {"preserve_thinking": True}}, max_tokens=2048, stream=True ) thinking_started = False has_thinking = False has_answer = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if delta.content: if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.3.3 Tool Calling Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment. ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] response = client.chat.completions.create( model="Qwen/Qwen3.6-35B-A3B-FP8", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, stream=True ) thinking_started = False has_thinking = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) if hasattr(delta, 'tool_calls') and delta.tool_calls: if has_thinking and thinking_started: print("\n=============== Content =================", flush=True) thinking_started = False for tool_call in delta.tool_calls: if tool_call.function: print(f"Tool Call: {tool_call.function.name}") print(f" Arguments: {tool_call.function.arguments}") if delta.content: print(delta.content, end="", flush=True) print() ``` # Qwen3.8 Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8 Deploy Qwen3.8 with SGLang — day-0 recipes for Qwen's 2.4T-parameter (95B active) hybrid GDN/GQA Mixture-of-Experts model on NVIDIA and AMD. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. **NVIDIA GPUs** (H200 / B200 / B300 / GB300) — the launch image, since this is a day-0 model with no release cut yet: ```bash Command theme={null} docker pull lmsysorg/sglang:qwen38 ``` **AMD GPUs** — pinned `v0.5.17` builds. The two are **not** interchangeable: they target different GPU architectures *and* different ROCm versions, so pick the one matching your hardware or AITER's kernels won't load. MI350X / MI355X (CDNA4, gfx950 — ROCm 7.20): ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260812 ``` MI300X (CDNA3, gfx942 — ROCm 7.00): ```bash Command theme={null} docker pull lmsysorg/sglang-rocm:v0.5.17-rocm700-mi30x-20260813 ``` For how to launch either image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with whatever the command generator below produces. Pick your hardware + quantization to generate the launch command. Three of the strategies are operating points on the throughput/latency curve; the fourth swaps the speculative decoder: * **Low Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving. * **High Throughput** — most tokens per second across many users. Best for batch jobs. * **DSpark** — the trained [DSpark draft model](#3-3-dspark-and-replayssm-speculative-decoding) instead of the checkpoint's built-in MTP head. Only offered where DSpark's constraints allow it (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)). ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The knobs come in two flavors: * **Built-in SGLang features** — TP / DP-Attention, MoE backend + EP (including WideEP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. * **Qwen3.8 specific features** — the **DSpark** speculative-decoding preset and its **ReplaySSM** opt-out, plus the GDN radix-cache-strategy and KV-cache-precision knobs (see [Configuration Tips](#2-configuration-tips) and [3.3 DSpark and ReplaySSM](#3-3-dspark-and-replayssm-speculative-decoding) below). ## 1. Model Introduction **Qwen3.8** (`Qwen3.8-2.4T-A95B`) is Qwen's largest open-weight model to date: **2.4T total parameters, 95B active per token**. It continues the hybrid-attention design of the Qwen3.5 / Qwen3.6 series, scaled up to 92 layers. * **Hybrid Attention** — 23 repeats of `3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)`, so 69 linear-attention layers to 23 full-attention ones. Gated Attention runs 64 query heads over 4 KV heads at head dimension 256. This balances linear computational complexity against long-context modeling quality. * **GDN (Gated Delta Network)** — the linear-attention layers pair a State Space Model with causal convolution (CausalConv1d), 128 V heads and 16 QK heads at head dimension 128. A fixed-size recurrent state replaces the growing KV cache, so memory is `O(1)` per layer while compute stays `O(N)`. * **Sparse Mixture-of-Experts** — 512 experts, 10 routed plus 1 shared active per token, expert intermediate dimension 2048. Hidden dimension 8192, vocabulary 248,320. * **MTP** — the checkpoint ships multi-token-prediction weights trained with multiple steps. That is what the NEXTN speculative recipes on this page decode against. **License:** [Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE). **Context length:** 262,144 native, extensible to 1,010,000 tokens. **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`. Raising `presence_penalty` toward 2 curbs runaway repetition at some risk of language mixing. For agentic work Qwen suggests allowing 262,144 tokens of reasoning and 131,072 for the final response. **Resources:** Each precision is its own repo — [BF16](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B) · [FP8](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8) · [NVFP4, NVIDIA Blackwell (RadixArk)](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) · [MXFP4, AMD CDNA4 (Qwen)](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4). Speculative-decoding draft model: [`RadixArk/Qwen3.8-2.4T-A95B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark) (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)). ## 2. Configuration Tips Four cells are marked **Not Verified** — **GB300 BF16** and the three GB300 **DSpark** strategies. They have launch recipes but no completed validation run; every other cell on the page has been run, including B300 NVFP4 DSpark. **Weight size decides the topology.** At 2.4T parameters BF16 is ≈4.8TB, FP8 ≈2.4TB, NVFP4 ≈1.2TB. FP8 fits no single node here — not even B300, whose 8 × 288GB = 2.30TB misses by a hair — so every FP8 recipe is multi-node. Single-node means FP4: NVFP4 on B300, MXFP4 on MI355X/MI350X. B200 NVFP4 would fit one node but pipelines two, because \~25GB per GPU after weights is too little to serve against. BF16 does not fit 16 GPUs either, so its one recipe is TP32 across 8 GB300 nodes — the only platform where a flat TP32 stays on rack-scale NVLink. **Two distinct FP4 checkpoints.** [NVFP4](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) is Blackwell-only; [MXFP4](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4) is MI350X/MI355X-only and hybrid (MXFP4 experts, FP8 attention/dense). MI300X is CDNA3 with no hardware MX matmul, so it serves FP8. Leave `--moe-runner-backend` unset on both and the runner resolves from the checkpoint's own `quant_method` — except the NVFP4 wide-EP tier, which pairs `flashinfer_trtllm_routed` with `--moe-a2a-backend flashinfer` by hand because auto cannot resolve that combination. ### GDN state is the scarce resource, not KV Two thirds of the layers are Gated DeltaNet, and their recurrent state lives in its own pool. That pool, not KV, is usually what caps concurrency — and a request's cost depends on the caching strategy:
Strategy State slots per request
--disable-radix-cache 1
no\_buffer 3
extra\_buffer (this model's auto) 5, or 4 where PP disables the overlap scheduler
So `--max-mamba-cache-size` has to match the ratio in force, or it silently clamps `max_running_requests` to a fraction of the target — every cell leaves the pool to `--mamba-full-memory-ratio` except GB300 FP8 Balanced and Low Latency, whose pins are part of tuned capacity sets — Low Latency's `--max-mamba-cache-size 80` is exactly its 16 concurrent requests × 5 slots. And `extra_buffer` needs radix caching on: `mamba_extra_buffer_of()` requires `disable_radix_cache` false, so adding `--disable-radix-cache` makes the strategy inert and drops the budget to one slot. ### NEXTN caps concurrency at 48 MTP weights ship inside the checkpoint, so NEXTN needs no draft model and the 3/1/4 preset fills in automatically. But a speculative cell with no `--max-running-requests` gets **48** from the speculative hook rather than a memory-derived ceiling — pin it explicitly to serve more. `pp_size > 1` rules speculative decoding out entirely in aggregated serving, which is why the H200, B200/B300 FP8, B200 NVFP4 and MI300X recipes carry no MTP. ### Linear-attention backends differ by GPU generation `--mamba-ssm-dtype bfloat16` is load-bearing on SM100: the flashinfer GDN decode default is gated on it, and without it decode silently falls back to Triton. On SM90 the GDN default is Triton for *both* halves, which is why H200 is the one cell pinning `--linear-attn-decode-backend flashinfer` too. The flashinfer GDN prefill default only covers chunk sizes up to 8192, so any cell with a larger `--chunked-prefill-size` must state `--linear-attn-prefill-backend flashinfer` itself. `--attention-backend trtllm_mha` is SM100-only. On Blackwell cells that leave it unset, the model hook picks it together with `--page-size 64` — and returns early when the backend *is* named, so an explicit backend also drops that paired page size and nothing then depends on `--speculative-eagle-topk` to keep the backend off Triton. ### GB300 tiers The FP8 ladder spans the whole curve on 4 nodes × 4 GPUs:
Tier Shape Spec
Low Latency TP16 narrow EP NEXTN 3+1
Balanced DP4×TP4 + EP16 NEXTN 3+1
High Throughput DP4×TP4 + EP16 off
Balanced and High Throughput share one shape and differ only in capacity. Low Latency is the odd one out: narrow EP wins at low concurrency, where spending ranks on expert parallelism costs more than it returns. MTP is off at saturation because draft-plus-verify overhead outweighs the speedup. NVFP4 has only two tiers — its recipes span different GPU counts (8 vs 16) and the wide-EP one holds capacity fixed across its whole concurrency list, so no third operating point exists. Extra build requirements: FP8 Balanced and High Throughput need the **DeepEP v2 wheel** (`2.1.0+01dc3aa`), since their `--moe-a2a-backend deepep_v2` flags do not exist upstream. NVFP4 High Throughput needs `nvfp4_agg_wideep_dep16_flashinfer_setup.sh` run first, and its `SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8192` is not optional — unset it falls back to 1024 and startup raises once `1024 × ep_size` no longer covers the largest CuteDSL MoE forward. ### GB300 PD disaggregation layouts The PD role selector adds the role and transfer flags to the selected base recipe. It does not resize the prefill and decode workers. Use separate workers with the layouts below for the measured GB300 operating points: | Checkpoint and operating point | Prefill workers | Decode worker | Capacity setting | | ------------------------------ | --------------- | --------------------------------------------------- | ------------------------------------------------------------------------ | | FP8, high throughput | 2 × TP1 / PP16 | DP4-attention / TP4 / EP16, DeepEP v2 + EPLB | Keep frontend concurrency above the decode MRR so prefill remains queued | | NVFP4, high throughput | 2 × TP1 / PP6 | DP2-attention / TP4 / EP8, FlashInfer one-sided A2A | Prefill MRR 128 per worker; decode MRR 512; frontend concurrency 1536 | | NVFP4, low latency | TP4 / PP2 | TP16 | Decode MRR 1 and frontend concurrency 1 at the latency endpoint | For NVFP4 with NEXTN, use the 3/1/4 settings on both roles so the prefill worker transfers the draft state. Enable ReplaySSM on the decode role. The generated router command sets the main policy to `round_robin`, which applies to prefill, and keeps the decode policy explicit; the server-side `--load-balance-method` does not configure router worker selection. Round robin is the measured choice for the fixed-shape throughput runs above. For agentic workloads with substantial repeated-prefix reuse, consider cache-aware routing, especially on the prefill side, and validate the cache-locality versus load-balance tradeoff on the target workload. ### AllReduce fusion The four cells running one flat TP group — GB300 FP8 Low Latency, GB300 NVFP4 Low Latency, GB300 BF16, B300 NVFP4 — set `SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION`, the Qwen3.5 CuteDSL path whose single workspace fuses AllReduce + Residual + RMSNorm with the MoE finalize. Worth 6–13% over the legacy path. Don't pass `--flashinfer-allreduce-fusion-backend` alongside it — the env suppresses the flag with a warning. Nothing else can use it: the fusion needs DP-attention off and the built-in TP MoE, so the wide-EP tiers are out, and the pipelined cells put their cross-node traffic on IB rather than NVLink. The three GB300 cells also carry `NCCL_NVLS_ENABLE=1`, because SGLang forces NVLS collectives off when that variable is unset. ### AMD Recommended: **MI355X + MXFP4, single node, TP8**. MI350X emits the identical command (same gfx950, same 288GB, same `mi35x` image); MI300X is CDNA3, takes the `mi30x` image, and needs two nodes for the FP8 weights. * **`--mem-fraction-static` looks aggressive on purpose.** With the aiter backend above 8192 context SGLang multiplies it by **0.85** before allocating, so MI355X's `0.9` lands at ≈0.765 and MI300X's `1.0` at ≈0.85. Don't "fix" these downward. MI300X must stay at 1.0 or the weights stop fitting. * **`--disable-custom-all-reduce` belongs on every MI300X rank** — SGLang resolves it per process, so setting it on one node would leave the two pipeline stages reducing through different code paths. * MI300X runs `--kv-cache-dtype fp8_e4m3` with `--page-size 16`: at 8 × 192GB per node the shape is memory-bound. ### ReplaySSM A GDN layer's recurrent state overwrites itself every token, so speculative verify has to be rewindable. Snapshotting the whole K×V state per draft step costs 64 KiB per request, layer and head at K=V=128, times γ+1 steps — scratch taken out of the same budget as the persistent state pool. [ReplaySSM](https://tridao.me/blog/2026/replayssm/) stores each draft step's raw inputs `Sᵢ = (vᵢ, kᵢ, gᵢ, βᵢ)` instead, a few hundred bytes written by the verify kernel on its way through. Once the sampler fixes the accepted length, one fold kernel replays the accepted prefix from the committed checkpoint and advances it in place. The fold is a verbatim clone of the verify recurrence, so the rebuilt state is bit-identical to the recurrent baseline; draft-step scratch shrinks by roughly two orders of magnitude and is never allocated. Folding on every commit is what lets it compose with radix prefix caching over the mutable GDN state: every `--mamba-track-interval` tokens the state is handed to the radix tree, and under `extra_buffer` it goes to a second slot so the running request keeps mutating its own. It is off by default — see [3.3](#3-3-dspark-and-replayssm-speculative-decoding). ## 3. Advanced Usage The `model` argument in the examples below is the BF16 repo id. Every precision is a **separate repo**, so `model` has to be the checkpoint the server was actually launched with — `…-A95B-FP8`, `…-A95B-NVFP4`, or `…-A95B-FP8-MXFP4`. The Deploy panel's cURL snippet always shows the right id for the cell you have selected. ### 3.1 Reasoning Qwen3.8 **always** reasons — thinking cannot be turned off, and every response opens with a `…` block. The `qwen3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) splits that block into `reasoning_content`, leaving `content` as the answer alone. Depth is tunable per request with `reasoning_effort` — `xhigh` (the default), `medium`, or `low`. `preserve_thinking` carries reasoning from earlier turns into context and is on by default. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="Qwen/Qwen3.8-2.4T-A95B", messages=[{"role": "user", "content": "What is 15% of 240?"}], reasoning_effort="xhigh", # xhigh (default) | medium | low ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Pending update — a sample transcript will be added here. ``` ### 3.2 Tool Calling Enable the `qwen3_coder` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="Qwen/Qwen3.8-2.4T-A95B", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Pending update — a sample transcript will be added here. ``` ### 3.3 DSpark and ReplaySSM (Speculative Decoding) We trained a **DSpark** draft model for Qwen3.8 with SpecForge. Turn it on with the **DSpark** chip in the **Speculative Decoding** card of the [Playground above](#playground) — it emits: ```bash Command theme={null} --speculative-algorithm DSPARK \ --speculative-draft-model-path RadixArk/Qwen3.8-2.4T-A95B-DSpark ``` **ReplaySSM is a separate opt-in.** `--enable-linear-replayssm-spec` defaults off and DSpark does not turn it on, so add it with the **ReplaySSM (spec)** row in the flag-select list. With it on (see [Configuration Tips](#2-configuration-tips) above for how it works), the verify kernel stores each draft step's raw inputs instead of snapshotting the full K×V GDN state, and a single fold kernel replays the accepted prefix from the last committed checkpoint. It's a pure side channel behind a ring buffer — the verify output is bitwise unchanged — so there's no accuracy tradeoff, only a memory one. **DSpark does not compose with every cell.** `_handle_dspark` rejects the run outright rather than degrading, so check these before turning the chip on: * **`--pp-size` must be 1.** The H200, B200, B300 (FP8) and MI300X cells are all pipelined, so DSpark is unavailable on them. * **With DP-Attention it additionally requires `--enable-dp-lm-head`**, the built-in TP MoE (`--moe-a2a-backend none`), and no context parallel. That rules out the GB300 wide-EP tiers, which run DP attention over DeepEP v2 or FlashInfer A2A. * `--speculative-num-steps` is forced to 1, and an omitted `--speculative-draft-model-path` only works if the target checkpoint bundles the draft weights. The Playground greys the DSpark chip out on the combinations above. The **DSpark** strategy chip in the Deploy panel emits this substitution on the four hw × quantization combinations that clear those constraints: GB300 FP8 (on the low-latency shape — the balanced tier's DeepEP v2 a2a rules DSpark out), GB300 NVFP4, GB300 BF16 and B300 NVFP4. Everything else on the page is either pipelined or wide-EP. Because the draft model needs its own weights and KV, those cells run tighter than their NEXTN counterparts — the validated B300 recipe drops `--mem-fraction-static` to 0.80 and trims `--context-length` to 200000 to buy the room back. ReplaySSM composes with radix prefix caching, overlap scheduling, and PD decode, so DSpark speculative decoding runs alongside the rest of the stack (WideEP, PD disaggregation, HiCache) rather than requiring any of them to be turned off. # Qwen3.8-27B Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B Deploy Qwen3.8-27B with SGLang — dense hybrid GDN vision-language model with BF16/FP8/NVFP4 W4A4 checkpoints and in-checkpoint MTP, single-GPU on H200, RTX PRO 6000, RTX 5090 and DGX Spark. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash Command theme={null} pip install --upgrade pip pip install uv git clone https://github.com/sgl-project/sglang.git && cd sglang git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e uv pip install --prerelease=allow -e "python[all]" ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-qwen38-27b-dflash2 ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point. `--mamba-full-memory-ratio` is the one sizing flag that matters for throughput on hybrid GDN models: the default (0.9) over-provisions the KV pool and silently clamps concurrency. Set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator) below; everything else follows the panels, and the computed value is pinned into the command. The RTX 5090 and RTX PRO 6000 cells above — including every Speculative Decoding / Serving Strategy / SSM dtype combination — were validated at ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark cells cover the full combination set — DFLASH2 included — re-measured end to end on `1cf2b8c`, but to a weaker standard: each of the 48 was confirmed to **boot and serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or acceptance-length numbers taken. ### Mamba ratio calculator Hybrid GDN models split post-weight memory into a worst-case-reserved **GDN state pool** (sets the concurrency ceiling) and a paged **attention KV pool**, divided by `--mamba-full-memory-ratio`. Every parameter below except `L` and the target concurrency is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio: ```text Formula theme={null} ratio = (S + D) x state_bytes / (L x kv_bytes_per_token) ``` * `S` — state slots per running request: `extra_buffer=5` (default), `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. For the two `extra_buffer` strategies, `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` frees one slot, and `extra_buffer` frees one more with the overlap scheduler off; the calculator reads both knobs. * `D` — verify intermediate states under speculative decoding: `--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4) and for DFLASH (8, DFlash2's block size); `--speculative-dspark-block-size + 1` for DSPARK, where the block size falls back to the draft checkpoint's `block_size` when the flag is omitted (7 for `RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with `--enable-linear-replayssm-spec`, which keeps the verify intermediates on a fixed ring instead of per-request slots. * `state_bytes` — one state slot, from the fixed geometry (48 GDN layers x 48 heads x 128 x 128 at `--mamba-ssm-dtype`, plus bf16 conv state): 153.9 MB at fp32, 78.4 MB at bf16. * `kv_bytes_per_token` — 16 attention layers x GQA 4 x 256 x K+V: 32.8 KB at fp8, 65.5 KB at bf16. * `L` — average total request length in tokens: input + output. `--max-mamba-cache-size = target_concurrency x S` is the equivalent explicit pin and overrides the ratio; the calculator emits it alongside. `D` is not a term here: the engine divides the state pool by `S` alone and sizes the speculative verify buffer separately, so folding `D` into the pin would over-provision the pool. After boot, verify with the `max_running_requests` line in the server log — it should not be capped below your target concurrency. ## Playground The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction **Qwen3.8-27B** is a dense hybrid Gated Delta Networks (GDN) **vision-language** model: a 27B causal language model paired with a vision encoder, with native image and video understanding alongside text. SGLang serves it through the Qwen3-VL path, so the vision tower is live on the recipes below. The language model is 64 layers, laid out as 16 repeats of *3 × (Gated DeltaNet → FFN)* followed by *1 × (Gated Attention → FFN)* — 48 linear-attention layers to 16 full-attention ones. Gated DeltaNet runs 48 value heads and 16 QK heads at head\_dim 128; Gated Attention is GQA 24/4 at head\_dim 256 with a 64-dim rotary slice. Hidden size is 5120 over a 17,408-dim FFN, and the checkpoint ships an MTP head trained with multiple steps. Context is 262,144 tokens natively, extensible to 1,000,000. The serving-relevant architecture is identical to Qwen3.6-27B. Thinking mode is on by default and can be disabled per request; reasoning depth is tunable with `reasoning_effort`, and `preserve_thinking` retains reasoning context from earlier messages.
Model Quantization Weights
Qwen3.8-27B BF16 Qwen/Qwen3.8-27B
Qwen3.8-27B-FP8 FP8 (blockwise) Qwen/Qwen3.8-27B-FP8
Qwen3.8-27B-NVFP4 (FP4 head) NVFP4 W4A4 + FP8 projections, `lm_head` packed to FP4 RadixArk/Qwen3.8-27B-NVFP4
Qwen3.8-27B-NVFP4 (BF16 head) Same body, `lm_head` left dense in BF16 RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead
The two NVFP4 exports differ only in the `lm_head`: one packs it to FP4, the other leaves it dense in BF16. The dense head is \~1.7 GB larger on disk and \~3.2 GB larger at runtime, so it is the harder of the two to fit — every recipe on this page was measured against it, and the FP4-head cells reuse those pins unchanged. Both NVFP4 checkpoints declare `kv_cache_quant_algo: FP8`; SGLang's default `--kv-cache-dtype auto` honors it, so the KV pool runs in `fp8_e4m3` with the checkpoint's calibration scales automatically. ## 2. Configuration Tips * **SM120/SM121 (RTX PRO 6000 Blackwell, RTX 5090, DGX Spark)**: use `--attention-backend flashinfer`; `trtllm_mha` is SM100-only. MTP with the FlashInfer backend requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len` (newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`. On DGX Spark the 128GB is unified memory shared with the host CPU, so all three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe at `--mem-fraction-static 0.80` rather than a separate operating point. The one lower pin is the unified pool pricing the host's memory too: 0.85 of 128GB leaves \~8GB for the OS — exactly DGX OS earlyoom's SIGTERM threshold — and the first long prefill or boot-time graph capture dips under it and gets the scheduler killed with `exit code -15` and no traceback (`journalctl -u earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way, and which cells is margin noise; at 0.80 every cell served on every attempt. **Validated on SM121 / aarch64**: all 48 configurations (3 checkpoints x Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included) booted and served on GB10 at `1cf2b8c` at ISL 8192 / OSL 1024, concurrency 1. That is boot-and-serve coverage only — no throughput or acceptance-length numbers — and it includes the FlashInfer `plan` / `uniform_q_len` path above, which raised no arity error on that build. Three host quirks when reproducing on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no `nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for memory because it is unified with the CPU — gate a relaunch on `MemAvailable` in `/proc/meminfo` instead; and the BF16 checkpoint takes \~6.5 minutes just to load its 18 shards from NVMe, so budget \~10 minutes to READY before calling a boot hung. * **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path and its cell is greyed out. The H200 recipes use 32768-token prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill backend engages by default under them. `--attention-backend fa3` is a valid alternative, measured slightly faster at bs=1. * **MTP**: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` uses the in-checkpoint MTP head. (This recipe was originally documented with `NEXTN`, an alias of `EAGLE` — same algorithm.) * **DSpark**: the trained draft model is a separate checkpoint — add `--speculative-algorithm DSPARK --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark` (the Playground's Speculative Decoding card emits this pair). DSpark does **not** take `--speculative-num-draft-tokens`: its verify window is `--speculative-dspark-block-size` (gamma) **+ 1**, and gamma is auto-inferred from the draft checkpoint when the flag is omitted (7 for this checkpoint, so D = 8). That `D` is a term in the balanced ratio — `r = (S + D) x token_equiv / L`, where `token_equiv` is the state slot expressed in KV tokens, `state_bytes / kv_bytes_per_token` (4698 at fp32 state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher `--mamba-full-memory-ratio` than no-spec at the same `S`, and pinning a different gamma changes the ratio with it. MTP is the opposite case: with `--enable-linear-replayssm-spec` its draft intermediates move onto a fixed ring, so `D = 0` and the ratio returns to the no-spec value. The [calculator](#mamba-ratio-calculator) applies both rules. * **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add `--speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the draft's block size, and it is the `D` term in the ratio, same value as DSpark's). The selector projects candidates through the target `lm_head`, including quantized heads, so it runs on the NVFP4 checkpoint (whose head is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the RTX PRO 6000 BF16/FP8 cells boot and serve; all 12 DGX Spark DFLASH2 cells boot and serve on `1cf2b8c` with the selector folded into the draft CUDA graph; on H200 and GB300 those cells carry the **Final Verification In Progress** badge. The RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 every pin is re-measured against that commit, and the panel applies them automatically: DFlash2 at `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — the only cell on this page needing a smaller prefill chunk, because at 0.91 the pools fit but a 2048-token chunk's activations do not — DSpark at 0.88, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90. Whether float32 is available with a draft model depends on the `lm_head`: on the BF16-head export it is greyed out for both DSpark and DFlash2, since the dense head's \~3.2 GB leave no fp32 state pool that also clears prefill graph capture. The FP4-head export frees that headroom back — DSpark serves at 0.89 and DFlash2 High-Throughput at 0.895 with `--mamba-full-memory-ratio 10` overriding the balanced value — and only DFlash2 Low-Latency stays out of reach, where five fp32 slots and a full request's KV never coexist. bfloat16 remains the faster choice regardless: DFlash2 posts 4.92 ms median TPOT at an accept length of 4.29, the best result on this card. * **Hardware fit**: FP8 weights \~28.5GB (not serviceable beyond bs≤2 on 32GB cards); NVFP4 weights \~16.5GB (recommended for RTX 5090-class GPUs). * `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per request from 5 slots to 4 at no accuracy cost. On small-VRAM cards (RTX 5090 32GB) the state pool bounds concurrency long before KV does — prefer lowering `S` (lazy strategy, or `--disable-radix-cache` for S=1); the [calculator](#mamba-ratio-calculator) re-derives the ratio for the new `S`. The balanced ratio itself is VRAM-independent. * `--mamba-ssm-dtype`: the GDN state slot is **153.9 MB at `float32`** (the checkpoint's declared precision) and **78.4 MB at `bfloat16`**, so bf16 roughly halves the state pool and hands the difference to KV — measured on an RTX 5090 with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB cards it also decides whether a config fits at all: EAGLE needs `--mem-fraction-static 0.94` at fp32 but 0.92 at bf16. Speed is **not** a one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE: 152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1); measure both for your quantization. Treat `bfloat16` as an accuracy gate and validate it for your workload. On SM120 both precisions run the Triton linear-attn prefill path — the FlashInfer GDN prefill fast path gates on SM100, where its validated domain is in fact a bf16 state pool — so no dtype forces an extra flag here. One interaction to know: `--enable-linear-replayssm-spec` auto-selects fp32 state when `--mamba-ssm-dtype` is unset, and an explicit non-fp32 value logs a state-drift warning at boot. The SSM dtype row always emits the flag explicitly, so the bf16 + EAGLE cells run with that warning — accounted for in their validation. * `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk on hybrid GDN models, and 8192-token chunks stall them \~600ms at a time. 2048 keeps decode inter-token latency smooth under mixed load and also improves single-wave TTFT. ## 3. Agent Harnesses Agent harnesses drive the model through the OpenAI-compatible endpoint — or, for Claude Code, through SGLang's Anthropic-compatible one — so any of them works once three things line up. **The parsers ship in the command.** Every recipe above carries `--reasoning-parser qwen3 --tool-call-parser qwen3_coder`, because without them a harness receives tool calls as raw text instead of structured `tool_calls`. The **Parsers** card in the [Playground](#playground) is therefore an opt-out — both chips start on, and turning one off strips its flag. `qwen3_coder` is the right tool-call parser for this checkpoint: its chat template instructs the model to reply with an inner `` / `` block nested in ``, which is exactly what that parser decodes. The Hermes parser (`--tool-call-parser hermes`) reads a *different* payload — bare JSON inside `` — so pointing a Hermes-format harness at this model without switching the flag yields tool calls that never parse. `--reasoning-parser qwen3` matches the template's `enable_thinking` toggle, which defaults to on. **Endpoint and model id.** The base URL is `http://:30000/v1`. The `model` string a harness sends must equal the server's `--model-path` — the OpenAI `/v1/models` name defaults to it — unless you override it with `--served-model-name`, which is usually worth doing to keep harness configs short. SGLang also serves an Anthropic-compatible `/v1/messages`, which is what [§3.3](#3-3-claude-code) uses. It converts each request to the OpenAI shape, hands it to the same chat-serving path, and converts the response back — so the parser flags above apply there identically. **Auth.** `--api-key` is unset by default, so the server accepts unauthenticated requests. Harnesses that insist on a key can send any placeholder; set `--api-key` on the server if the endpoint is reachable beyond localhost. ### 3.1 OpenCode [OpenCode](https://opencode.ai/docs/providers/) reaches a self-hosted endpoint through a provider entry in `opencode.json`. Store the credential first — pick **Other**, give the provider an id, and enter any placeholder when the server has no `--api-key`: ```bash Command theme={null} opencode /connect ``` Then declare the provider in `opencode.json`: ```json Config theme={null} { "$schema": "https://opencode.ai/config.json", "provider": { "sglang": { "npm": "@ai-sdk/openai-compatible", "name": "SGLang (Qwen3.8-27B)", "options": { "baseURL": "http://localhost:30000/v1" }, "models": { "RadixArk/Qwen3.8-27B-NVFP4": { "name": "Qwen3.8-27B NVFP4" } } } } } ``` `npm` selects the transport — `@ai-sdk/openai-compatible` is the one for a plain OpenAI-shaped endpoint. `apiKey` is optional and takes a `"{env:VAR_NAME}"` reference rather than a literal. The `models` keys are the ids sent on the wire, so they must match the served model name. Confirm with `/models`. ### 3.2 Pi [Pi](https://pi.dev/docs/latest/custom-provider) (`@earendil-works/pi-coding-agent`) registers providers from an extension rather than a config file. ```javascript Extension theme={null} pi.registerProvider("sglang", { baseUrl: "http://localhost:30000/v1", api: "openai-completions", apiKey: "$SGLANG_API_KEY", models: [ { id: "RadixArk/Qwen3.8-27B-NVFP4", name: "Qwen3.8-27B", reasoning: true, input: ["text", "image"], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow: 262144, maxTokens: 32768, }, ], }); ``` `api: "openai-completions"` is what selects the OpenAI-compatible transport, and `apiKey` takes a `$ENV_VAR` reference rather than a literal. `contextWindow` is the checkpoint's native 262,144; set `maxTokens` to whatever output cap you want per turn. Confirm registration with `pi --list-models`. ### 3.3 Claude Code Claude Code speaks the Anthropic API, so it points at SGLang's `/v1/messages` rather than the OpenAI endpoint. Anthropic documents that routing Claude Code to non-Claude models through a gateway is **not supported**. The wiring below works because SGLang implements the Anthropic message format, but it sits outside what Claude Code is tested against — expect newer Claude Code features to degrade or fail. `ANTHROPIC_BASE_URL` is the server origin — Claude Code appends `/v1/messages` itself, so leave the `/v1` suffix off: ```bash Command theme={null} export ANTHROPIC_BASE_URL=http://localhost:30000 export ANTHROPIC_AUTH_TOKEN=placeholder ``` The two credential variables travel in different headers: `ANTHROPIC_AUTH_TOKEN` goes out as `Authorization: Bearer`, `ANTHROPIC_API_KEY` as `x-api-key`. Either satisfies a server started without `--api-key`; with `--api-key` set, pick the variable matching the header your server reads. A credential variable also takes precedence over a saved claude.ai login for that session. The same pair can live in a settings file instead, which persists across shells and wins over a shell export: ```json Config theme={null} { "env": { "ANTHROPIC_BASE_URL": "http://localhost:30000", "ANTHROPIC_AUTH_TOKEN": "placeholder" } } ``` Run `/status` in Claude Code to confirm which base URL and credential source the session picked up. ### 3.4 Hermes Agent [Hermes Agent](https://github.com/NousResearch/hermes-agent) (Nous Research, MIT) selects a self-hosted endpoint through its setup wizard or its config file. ```bash Command theme={null} hermes model # choose "Custom endpoint (self-hosted / VLLM / etc.)", then enter the # base URL, an API key (blank for a local server) and the model name ``` Equivalently, in `~/.hermes/config.yaml`: ```yaml Config theme={null} model: default: RadixArk/Qwen3.8-27B-NVFP4 provider: custom base_url: http://localhost:30000/v1 api_key: "" context_length: 262144 ``` For several endpoints at once, declare them under `providers:` and switch with `/model custom:` mid-session: ```yaml Config theme={null} providers: workstation: api: http://localhost:30000/v1 server: api: https://gpu-host.internal:30000/v1 key_env: SGLANG_API_KEY ``` # Dots3-Note Source: https://docs.sglang.io/cookbook/autoregressive/RedNote/Dots3-Note Deploy RedNote dots3.note with SGLang — a native multimodal omni model (MoE ViT + Whisper-derived audio encoder + native video flattening) on the dots3 hybrid MLA/SWA language model, with DSA and full-sharing MTP speculative decoding. ## Deployment dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/33829/head && git checkout FETCH_HEAD uv pip install --prerelease=allow -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-dots3-note ``` This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick the hardware and the checkpoint precision. The recipe runs on a single 8-GPU Hopper node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet. **Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config. **Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path. ## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations signed off on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. ## 1. Model Introduction dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input. * **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor. * **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path. * **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers. * **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head. **Available checkpoints:** * **BF16**: [dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev) * **FP8**: [dots-studio/dots3-note-prev-fp8](https://huggingface.co/dots-studio/dots3-note-prev-fp8) **Resources:** [Hugging Face (BF16)](https://huggingface.co/dots-studio/dots3-note-prev) · [Hugging Face (FP8)](https://huggingface.co/dots-studio/dots3-note-prev-fp8) · [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829) ## 2. Configuration Tips **Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck. **MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`). **Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode. **DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`. **CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit). **Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool. **Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below. ## 3. Advanced Usage ### 3.1 Native video input dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="dots3.note", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": "https://example.com/sample.mp4"}, }, {"type": "text", "text": "Summarize what happens in this video."}, ], } ], extra_body={ "video_config": { "seq": 131072, "audio_cap": 0.5, "audio_sr": 16000, "k_mode": "eval_ek", }, }, ) print(response.choices[0].message.content) ``` ```text Output theme={null} Pending update... ``` Per-request video preprocessing controls are grouped under `video_config` in `extra_body`:
Field Default Purpose
seq 131072 Total sequence budget used by the video flattener.
audio\_cap 1.0 Maximum fraction of the input budget assigned to audio; 0 disables audio processing.
audio\_sr 16000 Audio sample rate.
k\_mode eval\_ek Deterministic evaluation/sampling mode of the flattener.
These controls are request-scoped so that evaluation jobs with different context budgets can share one server. For example: `extra_body={"video_config": {"seq": 131072, "audio_cap": 0.5}}`. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget. A request may carry several videos, and videos can be mixed with image and audio parts. Each video is flattened independently under the same per-request budget, and the flattened frames and audio segments are spliced back at the position of their `video_url` part, so the modality ordering of the prompt is preserved. ### 3.2 Image and audio input Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format. `--enable-multimodal` is in every cell; the vision and audio towers run in-process, so no extra server is needed. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="dots3.note", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": {"url": "https://example.com/sample.jpg"}, }, {"type": "text", "text": "Describe this image."}, ], } ], ) print(response.choices[0].message.content) ``` ```text Output theme={null} Pending update... ``` ### 3.3 Tool Calling Toggle **Tool Call Parser** (`--tool-call-parser dots`) and **Reasoning Parser** (`--reasoning-parser dots`) in the **Parsers** card of the [Playground above](#playground). Structured tool calls then surface via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="dots3.note", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) print(resp.choices[0].message.tool_calls) ``` ```text Output theme={null} Pending update... ``` ### 3.4 Encoder/LLM Disaggregation (EPD) `Dots3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment: * **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers. * **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together. # Step-3.7-Flash (new) Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step-3.7-Flash ## 1. Model Introduction [Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) is a 198B-parameter Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding. Engineered for high-frequency production workloads, it activates approximately 11B parameters per token and supports a 256k context window with three selectable reasoning levels (low, medium, and high). The model is available in multiple quantization formats (BF16, FP8, NVFP4). Step-3.7-Flash is built for developers who need to scale agentic workflows that combine perception, search, and reasoning — from parsing massive financial reports in one pass, to running multi-step search loops with cross-source verification, to operating concurrent coding agents in high-throughput pipelines. ## 2. SGLang Installation Step-3.7-Flash is currently available in SGLang via Docker image install. ### Docker (NVIDIA) ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ## 3. Model Deployment This section provides deployment configurations optimized for different use cases. ### 3.1 Basic Configuration The Step-3.7-Flash series comes in one size with multiple quantization options. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4x, TP=4), B200/B300 (4x, TP=4), GB200/GB300 (4x, TP=4). * **NVFP4 Quantization**: NVFP4 provides the smallest memory footprint. Requires `--quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 --moe-runner-backend flashinfer_trtllm`. * **Trust Remote Code**: All Step-3.7-Flash variants require `--trust-remote-code` due to the custom model architecture. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Step-3.7-Flash supports image inputs alongside text. Here's a basic example: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=messages, max_tokens=2048, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Multi-Image Input Example:** Step-3.7-Flash can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=messages, max_tokens=2048, ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` #### 4.2.2 Reasoning Parser Step-3.7-Flash supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.7-Flash \ --tp 4 \ --trust-remote-code \ --reasoning-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` #### 4.2.3 Tool Calling Step-3.7-Flash supports tool calling capabilities. Enable the tool call parser: **Start sglang server:** ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.7-Flash \ --tp 4 \ --trust-remote-code \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI import json client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # 1. define tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"} }, "required": ["location"] } } } ] # 2. tool run def get_weather(location, unit="celsius"): return f"The weather in {location} is 22 {unit[0].upper()} and sunny." # 3. send first request print("--- Sending first request ---") response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=1.0, stream=False ) message = response.choices[0].message # 4. Handle Reasoning Content reasoning = getattr(message, 'reasoning_content', None) if reasoning: print("=============== Thinking =================") print(reasoning) print("==========================================") # 5. Handle Tool Calls if message.tool_calls: print("\nTool Calls detected:") history_messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, message ] for tool_call in message.tool_calls: print(f" Tool: {tool_call.function.name}") print(f" Args: {tool_call.function.arguments}") args = json.loads(tool_call.function.arguments) tool_result = get_weather(args.get("location"), args.get("unit", "celsius")) history_messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) print("\n--- Sending tool results ---") final_response = client.chat.completions.create( model="stepfun-ai/Step-3.7-Flash", messages=history_messages, temperature=1.0, stream=False ) print("=============== Final Content =================") print(final_response.choices[0].message.content) else: if message.content: print("=============== Content =================") print(message.content) ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark *Benchmark results will be added soon.* # Step3-VL-10B Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3-VL-10B ## 1. Model Introduction [Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment. Key highlights of Step3-VL-10B include: * **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger. * **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning. * **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks. * **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications. For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B). ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs. ### 3.2 Configuration Tips * **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required. * **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios. * **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) * [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) ### 4.2 Advanced Usage #### 4.2.1 Multi-Modal Inputs Step3-VL-10B supports image inputs. Here's a basic example with image input: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png" } }, { "type": "text", "text": "Read all the text in the image." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, max_tokens=2048, extra_body={"top_k": -1} ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example output:** ```text Output theme={null} Response costs: 5.89s Generated text: Auntie Anne's CINNAMON SUGAR 1 × 17,000               17,000 SUB TOTAL                    17,000 GRAND TOTAL                 17,000 CASH IDR                    20,000 CHANGE DUE                 3,000 ``` **Multi-Image Input Example:** Step3-VL-10B can process multiple images in a single request for comparison or analysis: ```python Example theme={null} import time from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url="http://localhost:30000/v1", timeout=3600 ) messages = [ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg" } }, { "type": "image_url", "image_url": { "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg" } }, { "type": "text", "text": "Compare these two images and describe the differences in 100 words or less." } ] } ] start = time.time() response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, max_tokens=2048, extra_body={"top_k": -1} ) print(f"Response costs: {time.time() - start:.2f}s") print(f"Generated text: {response.choices[0].message.content}") ``` **Example Output:** ```text Output theme={null} Response costs: 3.24s Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods). ``` #### 4.2.2 Reasoning Parser Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --reasoning-parser deepseek-r1 \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` **Streaming with Thinking Process:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True, extra_body={"top_k": -1} ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Example Output:** ```text Output theme={null} =============== Thinking ================= Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that. First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24 0, I need to multiply 0.15 by 240. Let me do that calculation. Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh, that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too . So all methods point to 36. I think that's the answer. =============== Content ================= To solve the problem "What is 15% of 240?" step by step: --- ### **Step 1: Understand the concept of percentage** A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**. --- ### **Step 2: Convert the percentage to a decimal** To use percentages in calculations, convert them to decimals by dividing by 100: $$ 15\% = \frac{15}{100} = 0.15 $$ --- ### **Step 3: Multiply the decimal by the given number** Now, multiply 0.15 (the decimal form of 15%) by 240: $$ 0.15 \times 240 = 36 $$ --- ### **Alternative Verification Methods** #### **Method A: Break into parts** - 10% of 240 = $ 0.10 \times 240 = 24 $ - 5% of 240 = $ 0.05 \times 240 = 12 $ - Add them: $ 24 + 12 = 36 $ #### **Method B: Use direct multiplication** - $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $ #### **Method C: Estimate using known percentages** - 20% of 240 = $ 0.20 \times 240 = 48 $ - 10% of 240 = $ 0.10 \times 240 = 24 $ - 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $ --- ### **Final Answer** $$ \boxed{36} $$ ``` **Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions. #### 4.2.3 Tool Calling Step3-VL-10B supports tool calling capabilities. Enable the tool call parser: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --reasoning-parser deepseek-r1 \ --tool-call-parser hermes \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` **Python Example (with Thinking Process):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Define available tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city name" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit" } }, "required": ["location"] } } } ] # Make request with streaming to see thinking process response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=0.7, stream=True, extra_body={"top_k": -1} ) # Process streaming response thinking_started = False has_thinking = False tool_calls_accumulator = {} for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Accumulate tool calls if hasattr(delta, 'tool_calls') and delta.tool_calls: # Close thinking section if needed if has_thinking and thinking_started: print("\n=============== Content =================\n", flush=True) thinking_started = False for tool_call in delta.tool_calls: index = tool_call.index if index not in tool_calls_accumulator: tool_calls_accumulator[index] = { 'name': None, 'arguments': '' } if tool_call.function: if tool_call.function.name: tool_calls_accumulator[index]['name'] = tool_call.function.name if tool_call.function.arguments: tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments # Print content if delta.content: print(delta.content, end="", flush=True) # Print accumulated tool calls for index, tool_call in sorted(tool_calls_accumulator.items()): print(f"Tool Call: {tool_call['name']}") print(f" Arguments: {tool_call['arguments']}") print() ``` **Example Output:** ```text Output theme={null} =============== Thinking ================= The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters: - location: required (string) - "Beijing" - unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it I should call the function with location="Beijing". =============== Content ================= Tool Call: get_weather Arguments: {"location": "Beijing"} ``` **Handling Tool Call Results:** ```python Example theme={null} # After getting the tool call, execute the function def get_weather(location, unit="celsius"): # Your actual weather API call here return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # Send tool result back to the model messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, { "role": "assistant", "content": None, "tool_calls": [{ "id": "call_123", "type": "function", "function": { "name": "get_weather", "arguments": '{"location": "Beijing", "unit": "celsius"}' } }] }, { "role": "tool", "tool_call_id": "call_123", "content": get_weather("Beijing", "celsius") } ] final_response = client.chat.completions.create( model="stepfun-ai/Step3-VL-10B", messages=messages, temperature=0.7, extra_body={"top_k": -1} ) print(final_response.choices[0].message.content) ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA B200 GPU (1x) * Model: stepfun-ai/Step3-VL-10B * Tensor Parallelism: 1 * sglang version: 0.5.8+ We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. #### 5.1.1 Latency-Sensitive Benchmark * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model stepfun-ai/Step3-VL-10B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 30.85 Total input tokens: 14120 Total input text tokens: 720 Total input vision tokens: 13400 Total generated tokens: 4220 Total generated tokens (retokenized): 4217 Request throughput (req/s): 0.32 Input token throughput (tok/s): 457.71 Output token throughput (tok/s): 136.79 Peak output token throughput (tok/s): 240.00 Peak concurrent requests: 2 Total token throughput (tok/s): 594.50 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3083.40 Median E2E Latency (ms): 2747.00 P90 E2E Latency (ms): 4574.50 P99 E2E Latency (ms): 5462.49 ---------------Time to First Token---------------- Mean TTFT (ms): 1327.69 Median TTFT (ms): 1341.01 P99 TTFT (ms): 1486.11 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.16 Median TPOT (ms): 4.17 P99 TPOT (ms): 4.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.17 Median ITL (ms): 4.18 P95 ITL (ms): 4.30 P99 ITL (ms): 4.38 Max ITL (ms): 8.24 ================================================== ``` #### 5.1.2 Throughput-Sensitive Benchmark * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model stepfun-ai/Step3-VL-10B \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * Result: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 976.52 Total input tokens: 1416949 Total input text tokens: 76949 Total input vision tokens: 1340000 Total generated tokens: 510855 Total generated tokens (retokenized): 510526 Request throughput (req/s): 1.02 Input token throughput (tok/s): 1451.02 Output token throughput (tok/s): 523.14 Peak output token throughput (tok/s): 20429.00 Peak concurrent requests: 103 Total token throughput (tok/s): 1974.16 Concurrency: 99.81 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 97463.22 Median E2E Latency (ms): 91872.75 P90 E2E Latency (ms): 118553.42 P99 E2E Latency (ms): 198445.56 ---------------Time to First Token---------------- Mean TTFT (ms): 94379.07 Median TTFT (ms): 87163.09 P99 TTFT (ms): 194871.41 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 5.89 Median TPOT (ms): 5.72 P99 TPOT (ms): 23.58 ---------------Inter-Token Latency---------------- Mean ITL (ms): 6.05 Median ITL (ms): 0.13 P95 ITL (ms): 0.56 P99 ITL (ms): 3.99 Max ITL (ms): 97551.06 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 MMMU Benchmark You can evaluate the model's accuracy using the MMMU dataset: * Model Deployment Command: ```shell Command theme={null} python -m sglang.launch_server \ --model stepfun-ai/Step3-VL-10B \ --host 0.0.0.0 \ --port 30000 \ --trust-remote-code ``` * Benchmark Command: ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` * Result: ```text Output theme={null} Benchmark time: 934.6179109360091 answers saved to: ./answer_sglang.json Evaluating... answers saved to: ./answer_sglang.json {'Accounting': {'acc': 0.667, 'num': 30}, 'Agriculture': {'acc': 0.367, 'num': 30}, 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30}, 'Art': {'acc': 0.467, 'num': 30}, 'Art_Theory': {'acc': 0.5, 'num': 30}, 'Basic_Medical_Science': {'acc': 0.367, 'num': 30}, 'Biology': {'acc': 0.3, 'num': 30}, 'Chemistry': {'acc': 0.467, 'num': 30}, 'Clinical_Medicine': {'acc': 0.567, 'num': 30}, 'Computer_Science': {'acc': 0.467, 'num': 30}, 'Design': {'acc': 0.567, 'num': 30}, 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30}, 'Economics': {'acc': 0.6, 'num': 30}, 'Electronics': {'acc': 0.567, 'num': 30}, 'Energy_and_Power': {'acc': 0.633, 'num': 30}, 'Finance': {'acc': 0.733, 'num': 30}, 'Geography': {'acc': 0.333, 'num': 30}, 'History': {'acc': 0.533, 'num': 30}, 'Literature': {'acc': 0.533, 'num': 30}, 'Manage': {'acc': 0.6, 'num': 30}, 'Marketing': {'acc': 0.767, 'num': 30}, 'Materials': {'acc': 0.6, 'num': 30}, 'Math': {'acc': 0.7, 'num': 30}, 'Mechanical_Engineering': {'acc': 0.333, 'num': 30}, 'Music': {'acc': 0.4, 'num': 30}, 'Overall': {'acc': 0.523, 'num': 900}, 'Overall-Art and Design': {'acc': 0.483, 'num': 120}, 'Overall-Business': {'acc': 0.673, 'num': 150}, 'Overall-Health and Medicine': {'acc': 0.513, 'num': 150}, 'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120}, 'Overall-Science': {'acc': 0.5, 'num': 150}, 'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210}, 'Pharmacy': {'acc': 0.6, 'num': 30}, 'Physics': {'acc': 0.7, 'num': 30}, 'Psychology': {'acc': 0.467, 'num': 30}, 'Public_Health': {'acc': 0.733, 'num': 30}, 'Sociology': {'acc': 0.433, 'num': 30}} eval out saved to ./val_sglang.json Overall accuracy: 0.523 ``` # Step-3.5-Flash Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3.5 ## 1. Model Introduction [Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms. This generation delivers comprehensive upgrades across the board: * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models. * **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters. * **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a 3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness. ## 2.SGLang Installation Step-3.5-Flash is currently available in SGLang via Docker image install. ### Docker (NVIDIA) ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ### Docker (AMD ROCm) ```bash Command theme={null} # For MI300X/MI325X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x # For MI350X/MI355X docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x docker run -it \ --device=/dev/kfd --device=/dev/dri \ --shm-size=32g \ --ipc=host \ --network=host \ --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ lmsysorg/sglang:v0.5.9-rocm700-mi30x bash # or mi35x for MI350X/MI355X ``` ## 3.Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. ### 3.2 Configuration Tips * **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4). * **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X. * **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled. * **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels. ## 4.Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 \ --reasoning-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # Enable streaming to see the thinking process in real-time response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=[ {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"} ], temperature=0.7, max_tokens=2048, stream=True ) # Process the stream has_thinking = False has_answer = False thinking_started = False for chunk in response: if chunk.choices and len(chunk.choices) > 0: delta = chunk.choices[0].delta # Print thinking process if hasattr(delta, 'reasoning_content') and delta.reasoning_content: if not thinking_started: print("=============== Thinking =================", flush=True) thinking_started = True has_thinking = True print(delta.reasoning_content, end="", flush=True) # Print answer content if delta.content: # Close thinking section and add content header if has_thinking and not has_answer: print("\n=============== Content =================", flush=True) has_answer = True print(delta.content, end="", flush=True) print() ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We are asked: "What is 15% of 240?" We need to solve step by step. Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240. Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly. Method 1: 10% of 240 = 240 * 0.10 = 24. 5% is half of 10%, so 5% of 240 = 24 / 2 = 12. Then 15% = 10% + 5% = 24 + 12 = 36. Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36. We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36. Thus, the answer is 36. We'll present the solution step by step. =============== Content ================= To find 15% of 240, follow these steps: 1. **Convert the percentage to a decimal**: \( 15\% = \frac{15}{100} = 0.15 \) 2. **Multiply by the number**: \( 0.15 \times 240 = 36 \) Alternatively, break it down: - \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \) - \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%) - \( 15\% = 10\% + 5\% = 24 + 12 = 36 \) **Answer:** 36 ``` #### 4.2.2 Tool Calling Step-3.5 supports tool calling capabilities. Enable the tool call parser: **Python Example:** Start sglang server: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 \ --reasoning-parser step3p5 \ --tool-call-parser step3p5 ``` ```python Example theme={null} from openai import OpenAI import json client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) # 1. define tools tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "The city name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"} }, "required": ["location"] } } } ] # 2. tool run def get_weather(location, unit="celsius"): return f"The weather in {location} is 22°{unit[0].upper()} and sunny." # 3. send first request print("--- Sending first request ---") response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=[ {"role": "user", "content": "What's the weather in Beijing?"} ], tools=tools, temperature=1.0, stream=False ) message = response.choices[0].message # 4. Handle Reasoning Content reasoning = getattr(message, 'reasoning_content', None) if reasoning: print("=============== Thinking =================") print(reasoning) print("==========================================") # 5. Handle Tool Calls if message.tool_calls: print("\n🔧 Tool Calls detected:") history_messages = [ {"role": "user", "content": "What's the weather in Beijing?"}, message ] for tool_call in message.tool_calls: print(f" Tool: {tool_call.function.name}") print(f" Args: {tool_call.function.arguments}") args = json.loads(tool_call.function.arguments) tool_result = get_weather(args.get("location"), args.get("unit", "celsius")) history_messages.append({ "role": "tool", "tool_call_id": tool_call.id, "content": tool_result }) print("\n--- Sending tool results ---") final_response = client.chat.completions.create( model="stepfun-ai/Step-3.5-Flash", messages=history_messages, temperature=1.0, stream=False ) print("=============== Final Content =================") print(final_response.choices[0].message.content) else: if message.content: print("=============== Content =================") print(message.content) ``` **Output Example:** ```text Output theme={null} --- Sending first request --- =============== Thinking ================= The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine). ========================================== 🔧 Tool Calls detected: Tool: get_weather Args: {"location": "Beijing"} --- Sending tool results --- =============== Final Content ================= The weather in Beijing is 22°C and sunny. ``` **Note:** * The reasoning parser shows how the model decides to use a tool * Tool calls are clearly marked with the function name and arguments * You can then execute the function and send the result back to continue the conversation ## 5. Benchmark ### 5.1 Speed Benchmark **Test Environment:** * Hardware: NVIDIA H200 GPU (4x) * Model: Step-3.5-Flash * Tensor Parallelism: 4 * Expert Parallelism: 4 * sglang version: 0.5.8 We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. #### 5.1.1 Standard Scenario Benchmark * Model Deployment Command: ```shell Command theme={null} sglang serve \ --model-path stepfun-ai/Step-3.5-Flash \ --tp 4 \ --ep 4 ``` ##### 5.1.1.1 Low Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 35.30 Total input tokens: 6091 Total input text tokens: 6091 Total generated tokens: 4220 Total generated tokens (retokenized): 4212 Request throughput (req/s): 0.28 Input token throughput (tok/s): 172.57 Output token throughput (tok/s): 119.56 Peak output token throughput (tok/s): 124.00 Peak concurrent requests: 2 Total token throughput (tok/s): 292.14 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 3527.94 Median E2E Latency (ms): 2884.72 P90 E2E Latency (ms): 6350.38 P99 E2E Latency (ms): 7858.53 ---------------Time to First Token---------------- Mean TTFT (ms): 107.53 Median TTFT (ms): 80.93 P99 TTFT (ms): 269.52 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 8.12 Median TPOT (ms): 8.13 P99 TPOT (ms): 8.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 8.12 Median ITL (ms): 8.11 P95 ITL (ms): 8.61 P99 ITL (ms): 8.91 Max ITL (ms): 20.77 ================================================== ``` ##### 5.1.1.2 Medium Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 80 \ --max-concurrency 16 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 80 Benchmark duration (s): 54.06 Total input tokens: 39588 Total input text tokens: 39588 Total generated tokens: 40805 Total generated tokens (retokenized): 40479 Request throughput (req/s): 1.48 Input token throughput (tok/s): 732.33 Output token throughput (tok/s): 754.84 Peak output token throughput (tok/s): 928.00 Peak concurrent requests: 21 Total token throughput (tok/s): 1487.17 Concurrency: 14.06 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9501.23 Median E2E Latency (ms): 10010.71 P90 E2E Latency (ms): 15655.09 P99 E2E Latency (ms): 18803.63 ---------------Time to First Token---------------- Mean TTFT (ms): 198.34 Median TTFT (ms): 89.50 P99 TTFT (ms): 984.66 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 18.97 Median TPOT (ms): 18.80 P99 TPOT (ms): 35.67 ---------------Inter-Token Latency---------------- Mean ITL (ms): 18.27 Median ITL (ms): 17.48 P95 ITL (ms): 18.44 P99 ITL (ms): 62.47 Max ITL (ms): 460.85 ================================================== ``` ##### 5.1.1.3 High Concurrency * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model stepfun-ai/Step-3.5-Flash \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 500 Benchmark duration (s): 125.88 Total input tokens: 249331 Total input text tokens: 249331 Total generated tokens: 252662 Total generated tokens (retokenized): 251323 Request throughput (req/s): 3.97 Input token throughput (tok/s): 1980.77 Output token throughput (tok/s): 2007.23 Peak output token throughput (tok/s): 2500.00 Peak concurrent requests: 109 Total token throughput (tok/s): 3987.99 Concurrency: 92.25 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 23223.31 Median E2E Latency (ms): 22631.90 P90 E2E Latency (ms): 42269.38 P99 E2E Latency (ms): 47637.53 ---------------Time to First Token---------------- Mean TTFT (ms): 372.13 Median TTFT (ms): 127.26 P99 TTFT (ms): 1880.42 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 46.06 Median TPOT (ms): 47.61 P99 TPOT (ms): 51.34 ---------------Inter-Token Latency---------------- Mean ITL (ms): 45.31 Median ITL (ms): 39.86 P95 ITL (ms): 72.49 P99 ITL (ms): 117.05 Max ITL (ms): 1359.81 ================================================== ``` ### 5.2 Accuracy Benchmark #### 5.2.1 GSM8K Benchmark * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.few_shot_gsm8k --num-questions 200 ``` * **Results**: * Step-3.5-Flash ``` Accuracy: 0.885 Invalid: 0.005 Latency: 9.986 s Output throughput: 1972.911 token/s ``` # Hunyuan 3 Preview Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hunyuan3-Preview ## 1. Model Introduction Hunyuan 3 Preview (Hy3-preview) is Tencent's preview of its third-generation flagship MoE language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. **Key Features:** * **MoE Architecture**: 192 routed experts + 1 shared expert, 8 experts activated per token. \~276B total parameters with \~20B active, delivering dense-model quality at MoE inference cost. * **Hybrid Thinking**: Reasoning modes (`high`, `medium`, `low`, `none`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. * **Native Tool Calling**: Trained on structured `` / `` / `` grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. * **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. * **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. **Available Models:** * [tencent/Hy3-preview](https://huggingface.co/tencent/Hy3-preview) — BF16 instruct * [tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base) — BF16 base **Recommended Generation Parameters:**
Parameter Value
`temperature` 0.7
`top_p` 0.9
`reasoning_effort` `high` / `medium` / `low` (thinking) or `none` (instant)
**License:** TODO — verify on HuggingFace model card. ## 2. SGLang Installation SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. **Docker Images by Hardware Platform:**
Hardware Platform Docker Image
NVIDIA H200 / B200 / B300 / GB300 `lmsysorg/sglang:latest`
`lmsysorg/sglang:latest` bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime. For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization, and feature capabilities. ### 3.2 Configuration Tips **Key Parameters:**
Parameter Description Recommended Value
`--tool-call-parser` Tool call parser for function-calling support `hunyuan`
`--reasoning-parser` Reasoning parser for hybrid thinking modes `hunyuan`
`--trust-remote-code` Required for Hunyuan model loading Always enabled
`--mem-fraction-static` Static memory fraction (KV + activations) `0.9`
`--tp` Tensor parallelism size `2` / `4` / `8` depending on hardware
`--attention-backend` Attention backend (Blackwell only) `trtllm_mha`
`--speculative-algorithm` Speculative decoding via the bundled MTP draft `EAGLE` + `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`
**Hardware Requirements: NVIDIA BF16 (`Hy3-preview`, \~552GB weights)** * **H200 (141GB) / B200 (180GB)**: TP=8 (minimum for BF16 to fit single-node). * **B300 (275GB) / GB300**: TP=4. * **A100 / H100 (80GB)**: not supported single-node — BF16 requires multi-node TP=16+ on 80GB-class GPUs. **Blackwell (B200 / B300 / GB300):** Auto-selected attention backend can mis-route for HYV3 on Blackwell. Always pass `--attention-backend trtllm_mha` explicitly on Blackwell hardware (the config generator above enforces this). **Multi-Token Prediction (MTP):** The `Hy3-preview` release bundles an MTP draft module. SGLang runs it via its EAGLE speculative-decoding path — the draft module auto-loads from the same `--model-path`. Enable with the standard MTP flags: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.85 ``` Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload. **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation ### 4.1 Basic Usage For basic API usage and request examples, please refer to: * [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request) **Deployment Command (H200 × 8, BF16 default):** ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Testing Deployment:** After startup, you can test the SGLang OpenAI-compatible API with the following command: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "tencent/Hy3-preview", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ] }' ``` **Simple Completion Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"} ], max_tokens=1024 ) print("Reasoning:", response.choices[0].message.reasoning_content) print("Content: ", response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Reasoning: None Content: The Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in six games (4-2). This was the Dodgers' first World Series championship since 1988. The series was notable for being played in a neutral-site bubble at Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic. ``` When `reasoning_effort` is not set, the server defaults to instant mode (no thinking, `reasoning_content=None`). To opt into thinking, pass `reasoning_effort="high" / "medium" / "low"` on the request — see the Hybrid Thinking section below. ### 4.2 Advanced Usage #### 4.2.1 Reasoning Parser (Hybrid Thinking) Hy3-preview is a hybrid-thinking model. Control the thinking budget via the OpenAI-standard `reasoning_effort`: * `high` / `medium` / `low` — increasing amounts of chain-of-thought in `reasoning_content` * `none` — skip thinking entirely (instant responses, content-only) Enable the reasoning parser during deployment so that the thinking section (`...`) is separated into `reasoning_content`: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Thinking Mode — High Effort:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` **Output Example:** ```text Output theme={null} =============== Thinking ================= We need to solve: "What is 15% of 240?" Step by step. So we need to compute 15% of 240. The process: 15% means 15 per hundred, i.e., 15/100 = 0.15. Multiply 0.15 by 240. Or we can do: 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. Or do multiplication: 15/100 * 240 = (15*240)/100 = (3600)/100 = 36. So answer is 36. We need to produce step-by-step explanation. The instruction: "Solve step by step: What is 15% of 240?" So we should provide a clear solution with steps. The final answer: 36. Also maybe include units? No units. We'll output the solution in a clear manner. =============== Content ================= To find 15% of 240, follow these steps: 1. **Understand that percent means "per hundred."** So, 15% = 15/100 or 0.15. 2. **Multiply the number (240) by the percentage in decimal form.** 0.15 × 240. Alternatively, you can use fractions: (15/100) × 240. 3. **Perform the multiplication.** 0.15 × 240 = 36. Or: (15 × 240) / 100 = 3600 / 100 = 36. 4. **Check using an alternative method:** - 10% of 240 = 24. - 5% of 240 = half of 10% = 12. - 15% = 10% + 5% = 24 + 12 = 36. Thus, **15% of 240 is 36**. ``` **Instant Mode — No Thinking:** ```python Example theme={null} response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], reasoning_effort="none", max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` **Output Example:** ```text Output theme={null} Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. ``` #### 4.2.2 Tool Calling Hy3-preview supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips thinking tokens before the tool-call parser runs: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3-preview \ --tp 8 \ --reasoning-parser hunyuan \ --tool-call-parser hunyuan \ --trust-remote-code \ --mem-fraction-static 0.9 ``` **Non-Streaming Example:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example:** ```text Output theme={null} Reasoning: None Content: I'll get the current weather for Beijing in Fahrenheit for you. Tool Call: get_weather Arguments: {"city": "Beijing", "unit": "fahrenheit"} ``` **Streaming Example (incremental argument deltas):** Hy3-preview's `hunyuan` tool-call parser emits tool names first, then argument JSON in incremental fragments — matching the OpenAI streaming contract: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model="tencent/Hy3-preview", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, stream=True, ) tool_buffer = {} for chunk in stream: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) for tc in delta.tool_calls or []: buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) if tc.function and tc.function.name: buf["name"] += tc.function.name if tc.function and tc.function.arguments: buf["args"] += tc.function.arguments for idx, buf in tool_buffer.items(): print(f"\nTool[{idx}] {buf['name']}({buf['args']})") ``` **Output Example:** ```text Output theme={null} I'll check the current weather in Beijing for you using Fahrenheit. Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) ``` ## 5. Benchmark ### 5.1 Accuracy Benchmark **Test Environment:** * Hardware: 8× NVIDIA H200 (141GB) * Docker Image: `lmsysorg/sglang:hy3-preview` * Model: `tencent/Hy3-preview` (BF16) * Tensor Parallelism: 8 * SGLang version: latest `main` #### 5.1.1 GSM8K * Benchmark Method: 5-shot CoT on 200 questions, evaluated via SGLang native backend * Benchmark Command: ```bash Command theme={null} python3 benchmark/gsm8k/bench_sglang.py --num-questions 200 --parallel 64 ``` * Test Results: ```text Output theme={null} TODO — replace with real GSM8K accuracy after benchmark run on Hy3-preview (BF16). ``` #### 5.1.2 MMLU * Benchmark Method: 5-shot, all 57 subjects * Benchmark Command: ```bash Command theme={null} python3 benchmark/mmlu/bench_sglang.py --nsub 60 --parallel 64 ``` * Test Results: ```text Output theme={null} TODO — replace with real MMLU accuracy after benchmark run on Hy3-preview (BF16). ``` #### 5.1.3 Tool-Call Accuracy (MiniMax-Provider-Verifier) * Benchmark Tool: [MiniMax-Provider-Verifier](https://github.com/MiniMax-AI/MiniMax-Provider-Verifier) * Metric: function-call schema validity, argument match, and end-to-end response correctness * Test Results: ```text Output theme={null} TODO — replace with real tool-call accuracy after benchmark run on Hy3-preview (BF16). ``` ### 5.2 Speed Benchmark #### 5.2.1 Low Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model tencent/Hy3-preview \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 10 \ --max-concurrency 1 ``` * Test Results: ```text Output theme={null} TODO — replace with real low-concurrency output on Hy3-preview (BF16). ``` #### 5.2.2 High Concurrency * Benchmark Command: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model tencent/Hy3-preview \ --dataset-name random \ --random-input-len 1000 \ --random-output-len 1000 \ --num-prompts 500 \ --max-concurrency 100 ``` * Test Results: ```text Output theme={null} TODO — replace with real high-concurrency output on Hy3-preview (BF16). ``` # Hy3 Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy3 Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding. ## Deployment For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. ```bash theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate # Install from source (main carries the suffix-aware `hunyuan` parser + the # HYV3 model code). Once a tagged release picks it up, # `uv pip install --prerelease=allow sglang` is enough. git clone https://github.com/sgl-project/sglang.git cd sglang uv pip install --prerelease=allow -e python ``` Run the **Python** output of the command panel below in that environment. ```bash Command theme={null} # The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser. docker pull lmsysorg/sglang:dev ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces. The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding. Pick your hardware + recipe to generate the launch command. * **Low-Latency** — fastest reply for a single user. Pick for chat. * **Balanced** — good speed with several users at once. Use for typical multi-user serving.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, HF\_TOKEN, NODE\_RANK, NODE0\_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors: * **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers. * **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime). Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · \$ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you've actually run the command on your hardware and confirmed it works.
## 1. Model Introduction **Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. **Key Features:** * **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost. * **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning. * **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output. * **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning. * **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box. **Available Model:** * [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct * [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8 **Recommended Generation Parameters:**
Parameter Value
temperature 0.9
top\_p 1.0
reasoning\_effort high / low (thinking) or no\_think (instant)
**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `` instead of the bare ``). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model. ## 2. Configuration Tips **Hardware requirements (BF16, \~590GB weights):**
GPU VRAM TP Notes
H200 141GB 8 minimum single-node for BF16
B200 192GB 4 BF16 590GB → 148GB/GPU
B300 / GB300 288GB 4 BF16 590GB → 148GB/GPU; ample KV headroom
GB200 192GB 4 single-node 4×192GB = 768GB fits BF16 590GB
**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason. **MTP (Multi-Token Prediction, EAGLE).** * `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1. * `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient. **`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is. ## 3. Advanced Usage ### 3.1 Reasoning (Hybrid Thinking) Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`: * `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content` * `no_think` — skip thinking entirely (instant responses, content-only) Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}], reasoning_effort="high", max_tokens=2048, ) msg = response.choices[0].message print("=============== Thinking =================") print(msg.reasoning_content) print("=============== Content =================") print(msg.content) ``` ```text Output theme={null} =============== Thinking ================= We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240. 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36. =============== Content ================= To find 15% of 240, follow these steps: 1. 15% = 15/100 or 0.15. 2. Multiply 240 by 0.15: 0.15 × 240 = 36. 3. Check: 10% of 240 = 24, 5% = 12, 15% = 36. Thus, 15% of 240 is 36. ``` ```python Example theme={null} response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}], reasoning_effort="no_think", max_tokens=256, ) print("Content:", response.choices[0].message.content) ``` ```text Output theme={null} Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe. ``` ### 3.2 Tool Calling Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs: ```bash Command theme={null} sglang serve \ --model-path tencent/Hy3 \ --tp 8 \ --reasoning-parser auto \ --tool-call-parser auto ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city.", "parameters": { "type": "object", "properties": { "city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}, }, "required": ["city"], }, }, } ] response = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, ) msg = response.choices[0].message print("Reasoning:", msg.reasoning_content) print("Content: ", msg.content) for tc in msg.tool_calls or []: print(f"Tool Call: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` ```text Output theme={null} Reasoning: None Content: I'll get the current weather for Beijing in Fahrenheit for you. Tool Call: get_weather Arguments: {"city": "Beijing", "unit": "fahrenheit"} ``` ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") stream = client.chat.completions.create( model="tencent/Hy3", messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}], tools=tools, stream=True, ) tool_buffer = {} for chunk in stream: delta = chunk.choices[0].delta if delta.content: print(delta.content, end="", flush=True) for tc in delta.tool_calls or []: buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""}) if tc.function and tc.function.name: buf["name"] += tc.function.name if tc.function and tc.function.arguments: buf["args"] += tc.function.arguments for idx, buf in tool_buffer.items(): print(f"\nTool[{idx}] {buf['name']}({buf['args']})") ``` ```text Output theme={null} I'll check the current weather in Beijing for you using Fahrenheit. Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"}) ``` # Inkling Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling Deploy Inkling with SGLang — verified launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' 975B Mixture-of-Experts model with 1M-token context. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). Inkling has merged to `main` but isn't in a `pip` release yet — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. The Inkling images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

Panel controls (top of the command box):

  • ⧉ Copy — copies the current command to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, NODE\_RANK, NODE0\_IP) the command and cURL share.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **Inkling** is a Mixture-of-Experts model from Thinking Machines — **975B** total parameters, **41B** active per token, with a **1M-token** context window and **open weights** (BF16 and NVFP4 checkpoints below). It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache). **Resources:** HuggingFace — [Inkling](https://huggingface.co/thinkingmachines/Inkling) (BF16) · [Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4). ## 2. Configuration Tips **Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. **Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. **MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. **Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
reasoning\_effort value
none0.0
minimal0.1
low0.2
medium0.7
high0.9
xhigh0.99
max0.99
## 3. Advanced Usage ### 3.1 Reasoning Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "What is 17 times 24?"}], extra_body={"chat_template_kwargs": {"thinking": True}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user is asking for the product of 17 and 24. Let me calculate that. 17 × 24 I can break this down: 17 × 20 = 340 17 × 4 = 68 340 + 68 = 408 Alternatively: 24 × 10 = 240 24 × 7 = 168 240 + 168 = 408 So the answer is 408. Answer: 17 times 24 is **408**. Here's a quick breakdown: - 17 × 20 = 340 - 17 × 4 = 68 - 340 + 68 = **408** ``` ### 3.2 Tool Calling Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string", "description": "The city name"}}, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location. Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ### 3.3 Multimodal Input (Image + Audio) Inkling is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above). ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode() with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}, {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Describe the image, then transcribe the audio."}, ], } ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows. ### 3.4 LoRA (Serving Adapters) The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name. Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Option A (recommended): ":" in the model field resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4:lora0", messages=[{"role": "user", "content": "Summarize the changelog."}], ) # Option B: explicit lora_path via extra_body resp = client.chat.completions.create( model="thinkingmachines/Inkling-NVFP4", messages=[{"role": "user", "content": "Summarize the changelog."}], extra_body={"lora_path": "lora0"}, ) print(resp.choices[0].message.content) ``` One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence. ### 3.5 HiCache (Hierarchical KV Caching) Inkling serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`. ### 3.6 Long Context (MXFP8 KV) The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound. **Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200). The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality. To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300. ### 3.7 DSpark (Speculative Decoding) The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target. DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). # Inkling-Small Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-Small Deploy Inkling-Small with SGLang — launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' Inkling-Small Mixture-of-Experts model. ## Deployment For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). Inkling-Small has merged to `main` but isn't in a `pip` release yet — install from source: ```bash Command theme={null} pip install --upgrade pip pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python' ``` Then run the **Python** output of the command panel below. The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status. There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image: ```bash Command theme={null} docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

Panel controls (top of the command box):

  • ⧉ Copy — copies the current command to your clipboard.
  • \$ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST\_IP, PORT, NODE\_RANK, NODE0\_IP) the command and cURL share.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.
## Playground The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change. Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end. ## 1. Model Introduction **Inkling-Small** is a Mixture-of-Experts model from Thinking Machines with **open weights** (BF16 and NVFP4 checkpoints below), in the same architecture family as Inkling. It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling-Small on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache). **Resources:** HuggingFace — [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small) (BF16) · [Inkling-Small-NVFP4](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4). ## 2. Configuration Tips **Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving. **DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`. **Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage. **MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage. **Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):
reasoning\_effort value
none0.0
minimal0.1
low0.2
medium0.7
high0.9
xhigh0.99
max0.99
## 3. Advanced Usage ### 3.1 Reasoning Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "What is 17 times 24?"}], extra_body={"chat_template_kwargs": {"thinking": True}}, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Answer:", msg.content) ``` ```text Output theme={null} Reasoning: The user is asking for the product of 17 and 24. Let me calculate that. 17 × 24 I can break this down: 17 × 20 = 340 17 × 4 = 68 340 + 68 = 408 Alternatively: 24 × 10 = 240 24 × 7 = 168 240 + 168 = 408 So the answer is 408. Answer: 17 times 24 is **408**. Here's a quick breakdown: - 17 × 20 = 340 - 17 × 4 = 68 - 340 + 68 = **408** ``` ### 3.2 Tool Calling Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": {"location": {"type": "string", "description": "The city name"}}, "required": ["location"], }, }, } ] resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) msg = resp.choices[0].message print("Reasoning:", getattr(msg, "reasoning_content", None)) print("Content:", msg.content) print("Tool calls:", msg.tool_calls) ``` ```text Output theme={null} Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location. Content: Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)] ``` ### 3.3 Multimodal Input (Image + Audio) Inkling-Small is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above). ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") with open("image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode() with open("audio.wav", "rb") as f: audio_b64 = base64.b64encode(f.read()).decode() resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}}, {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}}, {"type": "text", "text": "Describe the image, then transcribe the audio."}, ], } ], max_tokens=1024, ) print(resp.choices[0].message.content) ``` Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows. ### 3.4 LoRA (Serving Adapters) The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name. Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`: ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") # Option A (recommended): ":" in the model field resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4:lora0", messages=[{"role": "user", "content": "Summarize the changelog."}], ) # Option B: explicit lora_path via extra_body resp = client.chat.completions.create( model="thinkingmachines/Inkling-Small-NVFP4", messages=[{"role": "user", "content": "Summarize the changelog."}], extra_body={"lora_path": "lora0"}, ) print(resp.choices[0].message.content) ``` One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence. ### 3.5 HiCache (Hierarchical KV Caching) Inkling-Small serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads. To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`. ### 3.6 Long Context (MXFP8 KV) The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound. **Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200). The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality. To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300. ### 3.7 DSpark (Speculative Decoding) The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark` — served unquantized alongside the NVFP4 target. DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). # MiMo-V2-Flash Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash ## Introduction XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware. This model creates a new balance between long-context modeling capability and inference efficiency. Key features include: * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias. * **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training. * **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length. * **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks. ## Installation MiMo-V2-Flash is currently available in SGLang via Docker image and pip install. ### Docker ```bash Command theme={null} # Pull the docker image docker pull lmsysorg/sglang:latest # Launch the container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash ``` ### Pip Installation ```bash Command theme={null} # On a machine with SGLang dependencies installed or inside a SGLang nightly container # Start an SGLang nightly container docker run -it --gpus all \ --shm-size=32g \ --ipc=host \ --network=host \ lmsysorg/sglang:latest bash # If you already have SGLang installed, uninstall the current SGLang version pip uninstall sglang -y # Install the PyPI Package pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \ --extra-index-url https://sgl-project.github.io/whl/pr/ ``` ## Model Deployment Use the configuration selector below to automatically generate the appropriate deployment command. MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X. ## Testing the deployment Once the server is running, test it with a chat completion request in another terminal: ```bash Command theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "XiaomiMiMo/MiMo-V2-Flash", "messages": [ {"role": "user", "content": "Hello! What can you help me with?"} ], "temperature": 0.7, "max_tokens": 100 }' ``` **Expected response:** ```json Config theme={null} { "id": "...", "object": "chat.completion", "model": "XiaomiMiMo/MiMo-V2-Flash", "choices": [{ "message": { "role": "assistant", "content": "Hello! I can help you with..." } }] } ``` ## Troubleshooting **DeepGEMM Timeout Error** Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast. **ROCm MI355X Attention Backend** If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`. # MiMo-V2.5 Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5 ## 1. Model Introduction [MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro) and [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5) are next-generation Mixture-of-Experts models from the XiaomiMiMo Team.
Variant Total params Active (MoE) Modalities
MiMo-V2.5-Pro 1.02T 42B Text (multimodal planned)
MiMo-V2.5 310B 15B Text, Image, Video, Audio
**Key Features:** * **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability. * **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding. Both variants support EAGLE speculative decoding with MTP weights. * **1M-Token Context**: Both variants support up to 1 million token context windows. * **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks. * **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API. **License:** Apache 2.0 ## 2. SGLang Installation Refer to the [official SGLang installation guide](../../../docs/get-started/install). **Docker Image:** All variants (MiMo-V2.5 310B and MiMo-V2.5-Pro 1.02T) use `lmsysorg/sglang:latest`, which ships CUDA 13.0 and runs on both Hopper (H100 / H200) and Blackwell (B200 / GB300). **TPU (sgl-jax):** MiMo-V2.5-Pro can also be served on TPU via the JAX-based [sgl-jax](https://github.com/sgl-project/sglang-jax) runtime. The container image and `pip install` steps are listed in [§3.3 TPU Deployment](#3-3-tpu-deployment-mimo-v2-5-pro-sgl-jax). ## 3. Model Deployment ### 3.1 Basic Configuration Use the selector below to generate the deployment command for your variant and hardware. ### 3.2 Configuration Tips **MiMo-V2.5-Pro (1.02T):** * **B200**: single node, TP=8 (verified). Uses `--attention-backend fa4` + `--moe-runner-backend flashinfer_trtllm` + `--mem-fraction-static 0.8`. Set `--swa-full-tokens-ratio 0.1` to keep KV-cache footprint within 192 GB HBM. * **GB300**: 2 nodes, TP=8 (verified). Same Blackwell stack as B200; multi-node interconnect requires `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`. Default SWA ratio is fine. * **H100/H200**: 2 nodes × 8 GPUs (TP=16, not yet verified). Uses the Hopper stack (`fa3` + DeepEP + EAGLE multi-layer); fits with `--mem-fraction-static 0.7` and `--swa-full-tokens-ratio 0.3`. DeepEP dispatch tuning: `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` avoids memory spikes during prefill. * EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `--enable-multi-layer-eagle` (both Hopper and Blackwell). See §5.4 for acceptance-rate behavior on natural text vs random prompts. **MiMo-V2.5 (310B):** * The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--dp = TP / 4`; for TP > 4 this also requires DP-attention. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--dp 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`. * Single-node deployments: H100/H200 8× GPUs (`--tp 8 --dp 2`), B200 4× GPUs (`--tp 4`, dp=1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization. * On Blackwell, pass `--attention-backend fa4`: MiMoV2's asymmetric KV (`head_dim` 192 / `v_head_dim` 128) fails on the SM100 default `trtllm_mha`, which requires equal K/V widths. * On Blackwell, pass `--mm-attention-backend fa4` for the V2.5 vision encoder. The checkpoint config requests FlashAttention-3 internally, but SGLang rejects FA3 on Blackwell and expects FA4 for multimodal attention. * On Blackwell, pass `--moe-runner-backend flashinfer_trtllm`; the default `auto` falls through to the triton fused-MoE runner, \~12% slower at bs=1 decode. * `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--enable-dp-attention` is on, to keep LM head and encoder sharding consistent. * EAGLE MTP uses the checkpoint's MTP weights. Enable with `--speculative-algorithm EAGLE` and `--enable-multi-layer-eagle` (both Hopper and Blackwell). * **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples. **DeepEP (optional toggle, Hopper-only):** * DeepEP replaces the default MoE all-to-all dispatch with a fused [DeepEP](https://github.com/deepseek-ai/DeepEP) backend; it lowers expert dispatch latency and memory traffic, so it pays off under **high concurrency / throughput-bound workloads** on H100/H200. Under concurrency=1 / latency-bound workloads the gain is negligible — leave it off. * Enabling adds `--moe-a2a-backend deepep` + `--moe-dense-tp-size 1` (and `--ep ` for Pro) plus `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` env to cap the dispatch buffer. Requires `pip install deep_ep` (not part of the default sglang install). * On Blackwell (B200, GB300) the verified MoE backend is `flashinfer_trtllm`; the DeepEP toggle is a no-op there. ### 3.3 TPU Deployment (MiMo-V2.5-Pro, sgl-jax) MiMo-V2.5-Pro can also be served on TPU via [sgl-jax](https://github.com/sgl-project/sglang-jax). The runtime is a separate JAX-based stack (`sgl_jax.launch_server`); pick **TPU v7x** or **TPU v6e** in the panel above to generate the launch command. Verified topologies: | TPU Type | Topology | Chips/Node | Nodes | Total Chips | JAX Devices/Chip | Total JAX Devices (= `--tp-size`) | | -------- | -------- | ---------- | ----- | ----------- | ---------------- | --------------------------------- | | **v7x** | 2×2×4 | 4 | 4 | 16 | 2 | 32 | | **v6e** | 4×4×4 | 4 | 16 | 64 | 1 | 64 | > v7x exposes **2 logical JAX devices per chip**, so `--tp-size = 16 chips × 2 = 32`. v6e exposes 1 device per chip, so `--tp-size = 64`. Always set `--tp-size` to the total JAX device count across all nodes, not the chip count. All nodes must sit in the same TPU slice and reach each other on the JAX init port (`20000`) and the TPU process port (`8471`). **Step 1 — Launch the JAX TPU container on every node:** ```shell Command theme={null} docker run -it --privileged \ --shm-size=32g \ --ipc=host \ --network=host \ -v /dev:/dev \ us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:jax0.8.1-rev1 bash ``` > The image is pinned to `jax0.8.1-rev1` to keep the JAX runtime aligned with sgl-jax's TPU extras. **Step 2 — Clone and install sgl-jax (inside the container):** ```shell Command theme={null} git clone https://github.com/sgl-project/sglang-jax.git cd sglang-jax pip install -e "python[tpu]" ``` ## 4. Model Invocation ### 4.1 Basic Usage See [Basic API Usage](../../../docs/basic_usage/send_request). ### 4.2 Reasoning Output Both variants support hybrid thinking mode. Thinking content is separated via the reasoning parser. **Thinking Mode (default):** ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[ {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."} ] ) print("====== Reasoning ======") print(response.choices[0].message.reasoning_content) print("====== Answer ======") print(response.choices[0].message.content) ``` **Output Example (MiMo-V2.5):** ```text theme={null} ====== Reasoning ====== Comparing 9.11 and 9.9. The integer parts are both 9. Now compare the decimal parts: 0.11 vs 0.9. 0.9 = 0.90, which is greater than 0.11. So 9.9 > 9.11. ====== Answer ====== **9.9 is larger than 9.11.** Here's the reasoning: When comparing decimals, line them up to the same number of decimal places: - 9.11 - 9.90 Both have a **9** in the ones place, but in the tenths place, **9 > 1**, so 9.90 > 0.11. **9.9 > 9.11** ``` **Thinking Off (instant mode):** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[ {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."} ], extra_body={"chat_template_kwargs": {"thinking": False}} ) print(response.choices[0].message.content) ``` **Output Example (MiMo-V2.5):** ```text theme={null} ## Comparing 9.11 and 9.9 **9.9 is larger.** The key is to compare them place by place. It helps to write them with the same number of decimal places: - **9.11** → 9.11 - **9.9** → 9.90 Both have **9** in the ones place, but in the tenths place: **9** (in 9.90) is greater than **1** (in 9.11). So **9.90 > 9.11**. ``` ### 4.3 Multimodal Invocation (V2.5 only) **Image Understanding:** ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"}}, {"type": "text", "text": "Describe this image in detail."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} Based on the image provided, here is a detailed description: The image captures a whimsical or surreal scene set on a busy city street, likely in New York City given the iconic yellow cabs. In the center foreground, a man is sitting on a folding chair, casually crossing his legs. He is wearing a bright yellow hoodie with a graphic on the front and blue jeans. He is intently focused on ironing a white dress shirt that rests on an ironing board set up directly on the asphalt. Behind him, a yellow SUV taxi cab is stopped or moving slowly, angled slightly away from the camera. To his left, another yellow taxi sedan is captured in motion blur, indicating it is driving past him. The background features tall city buildings with glass windows and storefronts. There are banners hanging from streetlights, and some greenery is visible in the distance. The overall impression is one of incongruity—performing a domestic chore like ironing in the middle of a chaotic urban environment. ``` **Video Understanding:** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "video_url", "video_url": {"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"}}, {"type": "text", "text": "Summarize what happens in this video."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} A person wearing blue protective gloves is shown operating a microscope in a close-up shot. The individual is adjusting a knob on the side of the microscope, which moves the stage holding a glass slide, likely focusing the lens on the specimen. ``` > Video decoding requires `decord` (`pip install decord`); SGLang's MiMo-V2.5 multimodal processor uses `decord.VideoReader` for frame extraction. **Audio Understanding:** ```python Example theme={null} response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{ "role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/audios/Trump_WEF_2018_10s.mp3"}}, {"type": "text", "text": "Transcribe and summarize this audio."} ] }] ) print(response.choices[0].message.content) ``` **Output Example:** ```text theme={null} **Transcript:** "Thank you Klaus very much. It's a privilege to be here at this forum where leaders in business, science, art, diplomacy and world affairs have gathered for..." **Summary:** The speaker thanks Klaus for the introduction and expresses their honor at attending a forum. They highlight that the event has brought together high-level leaders from various sectors, including business, science, art, and diplomacy. ``` ### 4.4 Tool Calling ```python Example theme={null} from openai import OpenAI client = OpenAI( base_url="http://localhost:30000/v1", api_key="EMPTY" ) tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a location", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City name"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]} }, "required": ["location"] } } } ] response = client.chat.completions.create( model="XiaomiMiMo/MiMo-V2.5", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools ) msg = response.choices[0].message if msg.reasoning_content: print("=== Reasoning ===") print(msg.reasoning_content) if msg.tool_calls: print("=== Tool Calls ===") for tc in msg.tool_calls: print(f" Function: {tc.function.name}") print(f" Arguments: {tc.function.arguments}") ``` **Output Example (MiMo-V2.5):** ```text theme={null} === Reasoning === The user wants to know the weather in Beijing. I have a function available called "get_weather" that can retrieve current weather for a location. Let me call that function with Beijing as the location. === Tool Calls === Function: get_weather Arguments: {"location": "Beijing"} ``` ## 5. Benchmark Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` with generated random prompts; text runs use 1024 input tokens and 1024 output tokens per request, and the image run uses 2 random 720p images per request. ### 5.1 Accuracy Benchmark #### 5.1.1 GSM8K Standard 5-shot, `temperature=0`, `max_tokens=4096`, model defaults to thinking-on (responses contain `...` and the eval extracts the trailing number via regex). Server launch: see [Section 3](#3-model-deployment). **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.run_eval \ --base-url http://127.0.0.1:30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --eval-name gsm8k \ --num-examples 200 \ --num-threads 8 \ --max-tokens 4096 \ --temperature 0.0 ``` > `run_eval.py` automatically appends `/v1` to `--base-url`; pass the bare `host:port` URL (without trailing `/v1`), otherwise requests resolve to `/v1/v1/chat/completions` and 404. * **Test Results:** * MiMo-V2.5-Pro (FP8, 8× B200) ``` Score: 0.965 (193 / 200) Latency: 253.90 s Output throughput: 461.78 tok/s ``` * MiMo-V2.5 (FP8, 8× H200) ``` Score: 0.980 (196 / 200) Latency: 477.52 s Output throughput: 88.9 tok/s ``` #### 5.1.2 MMMU (V2.5 only) `MMMU/MMMU` validation split (multi-discipline multimodal), `concurrency=16`, default sampling. * **Benchmark Command:** ```shell Command theme={null} python3 benchmark/mmmu/bench_sglang.py \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --concurrency 16 ``` * **Test Results:** * MiMo-V2.5 (FP8) ``` Pending update ``` ### 5.2 Speed Benchmark — MiMo-V2.5-Pro **Test Environment:** * Hardware: NVIDIA B200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`) * Recipe: Blackwell verified baseline (EAGLE off for this benchmark — see note below) * sglang version: 0.5.11 > The numbers in §5.2 are the **no-EAGLE baseline** on `random 1024/1024`. On uniform-random token streams the MiMo-V2.5-Pro 3-layer MTP draft has very low accept-rate (\~0.13–0.27 vs \~0.75 on natural-text prompts, see §5.4) — there's no token-co-occurrence signal for the draft to model — so EAGLE here adds verify overhead without recovering enough draft tokens to be a net win on this workload. EAGLE MTP itself works on B200 + `--enable-multi-layer-eagle` (see §3 deployment command and §5.4 for an acceptance profile on natural text). #### 5.2.1 Latency-Sensitive Benchmark * **Model Deployment Command:** see the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 27.59 Total input tokens: 1997 Total input text tokens: 1997 Total generated tokens: 2798 Total generated tokens (retokenized): 2794 Request throughput (req/s): 0.36 Input token throughput (tok/s): 72.38 Output token throughput (tok/s): 101.41 Peak output token throughput (tok/s): 110.00 Peak concurrent requests: 3 Total token throughput (tok/s): 173.79 Concurrency: 1.00 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2757.26 Median E2E Latency (ms): 3319.10 P90 E2E Latency (ms): 4157.47 P99 E2E Latency (ms): 4869.32 ---------------Time to First Token---------------- Mean TTFT (ms): 162.17 Median TTFT (ms): 68.11 P99 TTFT (ms): 929.58 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 9.19 Median TPOT (ms): 9.33 P99 TPOT (ms): 9.39 ---------------Inter-Token Latency---------------- Mean ITL (ms): 9.31 Median ITL (ms): 9.35 P95 ITL (ms): 9.44 P99 ITL (ms): 9.77 Max ITL (ms): 19.80 ================================================== ``` #### 5.2.2 Throughput-Sensitive Benchmark * **Model Deployment Command:** see the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 112.78 Total input tokens: 302118 Total input text tokens: 302118 Total generated tokens: 195775 Total generated tokens (retokenized): 191069 Request throughput (req/s): 8.87 Input token throughput (tok/s): 2678.83 Output token throughput (tok/s): 1735.90 Peak output token throughput (tok/s): 3040.00 Peak concurrent requests: 121 Total token throughput (tok/s): 4414.73 Concurrency: 87.80 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 9901.96 Median E2E Latency (ms): 6525.54 P90 E2E Latency (ms): 23567.98 P99 E2E Latency (ms): 42109.22 ---------------Time to First Token---------------- Mean TTFT (ms): 223.69 Median TTFT (ms): 139.45 P99 TTFT (ms): 1082.02 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 50.63 Median TPOT (ms): 51.66 P99 TPOT (ms): 91.41 ---------------Inter-Token Latency---------------- Mean ITL (ms): 49.79 Median ITL (ms): 33.69 P95 ITL (ms): 103.37 P99 ITL (ms): 151.34 Max ITL (ms): 1600.00 ================================================== ``` ### 5.3 Speed Benchmark — MiMo-V2.5 **Test Environment:** * Hardware: NVIDIA H200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5` (FP8) * Tensor Parallelism: 8 (DP-attention with `--dp 2`) * Recipe: Balanced (DP-attn + EAGLE MTP) * sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`) #### 5.3.1 Latency-Sensitive Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 14.72 Total input tokens: 1997 Total input text tokens: 1997 Total generated tokens: 2798 Total generated tokens (retokenized): 2697 Request throughput (req/s): 0.68 Input token throughput (tok/s): 135.67 Output token throughput (tok/s): 190.09 Peak output token throughput (tok/s): 245.00 Peak concurrent requests: 3 Total token throughput (tok/s): 325.77 Concurrency: 1.00 Accept length: 3.08 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 1469.98 Median E2E Latency (ms): 1652.84 P90 E2E Latency (ms): 2210.80 P99 E2E Latency (ms): 2823.86 ---------------Time to First Token---------------- Mean TTFT (ms): 143.89 Median TTFT (ms): 99.25 P99 TTFT (ms): 481.01 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 4.87 Median TPOT (ms): 4.30 P99 TPOT (ms): 6.64 ---------------Inter-Token Latency---------------- Mean ITL (ms): 4.76 Median ITL (ms): 3.46 P95 ITL (ms): 13.52 P99 ITL (ms): 13.84 Max ITL (ms): 74.37 ================================================== ``` #### 5.3.2 Throughput-Sensitive Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1000 \ --max-concurrency 100 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 100 Successful requests: 1000 Benchmark duration (s): 93.41 Total input tokens: 302118 Total input text tokens: 302118 Total generated tokens: 195775 Total generated tokens (retokenized): 188139 Request throughput (req/s): 10.71 Input token throughput (tok/s): 3234.48 Output token throughput (tok/s): 2095.97 Peak output token throughput (tok/s): 3019.00 Peak concurrent requests: 121 Total token throughput (tok/s): 5330.45 Concurrency: 91.04 Accept length: 2.95 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 8503.45 Median E2E Latency (ms): 7491.96 P90 E2E Latency (ms): 13706.99 P99 E2E Latency (ms): 20474.33 ---------------Time to First Token---------------- Mean TTFT (ms): 4399.20 Median TTFT (ms): 4333.35 P99 TTFT (ms): 8004.81 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 58.23 Median TPOT (ms): 21.78 P99 TPOT (ms): 747.79 ---------------Inter-Token Latency---------------- Mean ITL (ms): 20.06 Median ITL (ms): 15.28 P95 ITL (ms): 48.36 P99 ITL (ms): 96.99 Max ITL (ms): 969.61 ================================================== ``` #### 5.3.3 Multimodal (Image) Benchmark * **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment). * Benchmark Command: ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 30000 \ --model XiaomiMiMo/MiMo-V2.5 \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 \ --random-output-len 1024 \ --num-prompts 10 \ --max-concurrency 1 ``` * **Test Results:** ```text Output theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai-chat Traffic request rate: inf Max request concurrency: 1 Successful requests: 10 Benchmark duration (s): 25.73 Total input tokens: 661 Total input text tokens: 631 Total input vision tokens: 30 Total generated tokens: 4220 Total generated tokens (retokenized): 0 Request throughput (req/s): 0.39 Input token throughput (tok/s): 25.69 Output token throughput (tok/s): 164.03 Peak output token throughput (tok/s): 1.00 Peak concurrent requests: 2 Total token throughput (tok/s): 189.73 Concurrency: 1.00 Accept length: 2.94 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2570.74 Median E2E Latency (ms): 2411.92 P90 E2E Latency (ms): 3711.62 P99 E2E Latency (ms): 4949.74 ---------------Time to First Token---------------- Mean TTFT (ms): 0.00 Median TTFT (ms): 0.00 P99 TTFT (ms): 0.00 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 7.31 Median TPOT (ms): 6.17 P99 TPOT (ms): 17.18 ---------------Inter-Token Latency---------------- Mean ITL (ms): 0.00 Median ITL (ms): 0.00 P95 ITL (ms): 0.00 P99 ITL (ms): 0.00 Max ITL (ms): 0.00 ================================================== ``` ### 5.4 Multi-Layer EAGLE Acceptance Profile — MiMo-V2.5-Pro Pro's 3-layer MTP behaves very differently on natural text vs uniform-random token streams. The §5.2 benchmarks use `random 1024/1024`, which collapses accept-rate; this section measures the same server on GSM8K so the acceptance number is comparable to real workloads. **Test Environment:** * Hardware: NVIDIA B200 GPU (8×) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`) * Recipe: 3-layer EAGLE — `--enable-multi-layer-eagle --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` (top-1, max accept length 4) **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.test.run_eval \ --base-url http://127.0.0.1:30000 \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --eval-name gsm8k \ --num-examples 200 \ --num-threads 4 ``` The `accept_rate` and `accept_length` rows below are not part of `run_eval`'s own output — they were aggregated from the server-side `Decode batch ... accept rate: X accept len: Y` log lines emitted during the GSM8K run (307 batches total). | Workload | accept\_rate | accept\_length (max = 4) | | ------------------------------ | ------------ | ------------------------ | | GSM8K (natural text) | **0.755** | **3.27** | | `random 1024/1024` (reference) | 0.13–0.27 | \~1.x | GSM8K Score: **0.97** (194 / 200), output throughput ≈ 635 tok/s end-to-end on this single-server run. The accept-rate gap is intrinsic to MTP-style speculative decoding: the draft model is trained on natural-language token distributions and has no useful signal on uniform-random byte sequences. Workloads with structure (chat, code, reasoning traces) should expect the GSM8K-class number; the random-prompt baseline in §5.2 is a worst case for draft acceptance. ### 5.5 Long-Context Prefill & MTP Decode — MiMo-V2.5-Pro (Reference) Reference numbers from the [day0 enablement PR](https://github.com/sgl-project/sglang/pull/23808), collected on a 2-node Hopper deployment with the **EP=16, DP=2, TP=16** recipe (`--moe-a2a-backend deepep`, `--attention-backend fa3`, `--enable-multi-layer-eagle`). The setup, parallelism, and benchmark methodology all differ from §5.2 (Blackwell TP=8 with `random 1024/1024`), so treat these as a separate operating point — long-context prefill scaling and the MTP decode speedup — rather than a comparison against §5.2. **Test Environment:** * Hardware: NVIDIA Hopper GPU (2 nodes × 8 GPUs, GPU SKU intentionally not disclosed) * Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8) * Parallelism: `--tp 16 --dp 2 --ep 16 --moe-dense-tp-size 1 --enable-dp-attention` * Recipe: Hopper EP16 (DeepEP + EAGLE multi-layer MTP) #### 5.5.1 Long-Context Prefill Throughput Test setting: `chunked_prefill_size=32K`, `random_output_len=1`, cache flushed before every run. For input lengths ≥ 512K the workload was split into two requests routed to distinct DP ranks and the per-node throughput was read from `bench_serving` output. * **Benchmark Command:** ```shell Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --model XiaomiMiMo/MiMo-V2.5-Pro \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --random-input-len \ --random-output-len 1 \ --random-range-ratio 1.0 \ --flush-cache \ --seed 12345 \ --num-prompts 10000 ``` * **Test Results** — single-node prefill throughput, cache-miss: | Input length | Output length | Single-node prefill throughput | | ------------ | ------------- | ------------------------------ | | 4K | 1 | 30.80K tok/s | | 8K | 1 | 30.65K tok/s | | 16K | 1 | 29.85K tok/s | | 32K | 1 | 28.60K tok/s | | 64K | 1 | 26.65K tok/s | | 128K | 1 | 23.00K tok/s | | 256K | 1 | 17.90K tok/s | | 512K | 1 | 11.30K tok/s | | 768K | 1 | 9.40K tok/s | | 1M | 1 | 7.30K tok/s | Prefill throughput stays within \~10% of peak from 4K up to 32K and degrades gracefully past 128K, confirming the hybrid SWA+GA attention works correctly at 1M context. #### 5.5.2 Decode Throughput — MTP Speedup Test setting: fixed **16K input / 1K output**, varying batch size per DP rank, with and without the 3-layer MTP module. `MTP accept length` is the average number of draft tokens accepted per step under EAGLE speculative decoding. **TPS** below is per-request output tokens/sec (i.e. single-user perceived speed); the rightmost column is aggregated single-node decode throughput (= TPS × batch size). * **Test Results** — single-node decode throughput: | BS per DP rank | MTP | MTP accept length | Per-request TPS | Single-node decode throughput | | -------------- | -------- | ----------------- | --------------- | ----------------------------- | | 64 | disabled | - | 29.3 | 1875 tok/s | | 64 | 3-layer | 3 | 60.5 | 3873 tok/s | | 64 | 3-layer | 4 | 79.7 | 5103 tok/s | | 96 | disabled | - | 26.7 | 2564 tok/s | | 96 | 3-layer | 3 | 50.4 | 4840 tok/s | | 96 | 3-layer | 4 | 64.8 | 6225 tok/s | **Summary — MTP on / off:** | BS per DP rank | Without MTP | 3-layer MTP, accept=3 | 3-layer MTP, accept=4 | | -------------- | ----------- | --------------------- | --------------------- | | 64 | 1875 tok/s | 3873 tok/s (2.07×) | 5103 tok/s (2.72×) | | 96 | 2564 tok/s | 4840 tok/s (1.89×) | 6225 tok/s (2.43×) | The 3-layer MTP module delivers \~2× decode throughput at accept length 3 and \~2.5–2.7× at accept length 4 — the same order of magnitude as the "2–3× decode speedup" guidance in §3.2. # Overview Source: https://docs.sglang.io/cookbook/autoregressive/intro Practical guides for deploying and using large language models and vision language models with SGLang. # Autoregressive Model Benchmark Documentation Source: https://docs.sglang.io/cookbook/base/benchmarks/autoregressive_model_benchmark `sglang.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Large Language Models (LLMs) and Vision Language Models(VLMs). It supports various backends (`SGLang`, `vLLM`, etc.) and offers flexible configurations for request rates, dataset types, and profiling. ## 1. Quick Start ### Basic Usage (Random Data) Run a benchmark using randomly generated prompts with a local SGLang server. ```bash Command theme={null} python -m sglang.bench_serving --backend sglang --port 30000 --dataset-name random --num-prompts 100 ``` ### Real-World Data (ShareGPT) Run a benchmark using the ShareGPT dataset with a specific request rate. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --dataset-name sharegpt \ --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \ --num-prompts 1000 \ --request-rate 10 ``` ## 2. Parameter Reference ### 2.1 Backend & Server Configuration These parameters define the target server and the inference engine being used.
Parameter Description
`--backend` **Required.** Specifies the backend engine. Options: `sglang`, `sglang-native`, `sglang-oai`, `sglang-oai-chat`, `vllm`, `vllm-chat`, `lmdeploy`, `lmdeploy-chat`, `trt`, `gserver`, `truss`.
`--base-url` The API base URL (if not using specific host/port flags).
`--host` Server hostname. Default: `0.0.0.0`.
`--port` Server port. If not set, it defaults to the specific backend's standard port.
`--model` Model name or path. If unset, it queries `/v1/models` for configuration.
`--served-model-name` The model name used in the API request body. Defaults to the value of `--model`.
`--tokenizer` Path or name of the tokenizer. Defaults to the model configuration.
### 2.2 Dataset Configuration Controls the source of the prompts used for benchmarking.
Parameter Description
`--dataset-name` The type of dataset. Options: `sharegpt`, `custom`, `random`, `random-ids`, `generated-shared-prefix`, `mmmu`, `image`, `mooncake`.
`--dataset-path` File path to the dataset (e.g., local JSON file for ShareGPT).
`--num-prompts` Total number of prompts to process. Default: `1000`.
`--seed` Random seed for reproducibility.
`--tokenize-prompt` Uses integer IDs instead of strings for inputs. Useful for precise length control.
### 2.3 Input/Output Length Control Parameters to control the shape of requests (context length and generation length). #### For Random/Image Datasets: * `--random-input-len`: Number of input tokens per request. * `--random-output-len`: Number of output tokens per request. * `--random-range-ratio`: Range ratio for sampling input/output lengths. #### For ShareGPT Dataset: * `--sharegpt-output-len`: Overrides the output length defined in the dataset for each request. * `--sharegpt-context-len`: Max context length. Requests exceeding this are dropped. #### General Request Modifiers: * `--extra-request-body`: Appends a JSON object to the request payload (e.g., \{"key": "value"}). Useful for passing sampling parameters. * `--prompt-suffix`: A string suffix appended to all user prompts. * `--disable-ignore-eos`: If set, the model will stop generation upon hitting the EOS token (benchmarks usually ignore EOS to force max generation length). * `--apply-chat-template`: Applies the model's chat template to the input. ### 2.4 Traffic & Concurrency Controls how fast requests are sent to the server.
Parameter Description
`--request-rate` Requests per second (RPS). If `inf` (default), all requests are sent immediately (burst). Otherwise, arrival times follow a Poisson process.
`--max-concurrency` The maximum number of active requests allowed at once. Even if `request-rate` is high, the client will hold back requests if this limit is reached.
`--warmup-requests` Number of requests to run before the actual measurement begins to warm up the server.
`--flush-cache` Flushes the server cache before starting the benchmark.
### 2.5 Output & Logging
Parameter Description
`--output-file` Path to save the results in JSONL format.
`--output-details` Includes detailed metrics in the output.
`--print-requests` Prints requests to stdout as they are sent (useful for debugging).
`--disable-tqdm` Hides the progress bar.
`--disable-stream` Disables streaming mode (waits for full response).
`--return-logprob` Requests logprobs from the server.
`--tag` An arbitrary string tag added to the output file for identification.
### 2.6 Advanced #### 2.6.1 Image / Multi-modal Only applicable when --dataset-name is set to image. * `--image-count`: Number of images per request. * `--image-resolution`: Resolution (e.g., 1080p, 4k, or custom 1080x1920). * `--image-format`: jpeg or png. * `--image-content`: random (noise) or blank. #### 2.6.2 LoRA Benchmarking Used to simulate multi-LoRA serving scenarios. * `--lora-name`: A list of LoRA adapter names (e.g., `--lora-name` adapter1 adapter2). * `--lora-request-distribution`: How requests are assigned to adapters: * `uniform`: Equal probability. * `distinct`: New adapter for every request. * `skewed`: Follows a Zipf distribution (simulating hot/cold adapters). * `--lora-zipf-alpha`: The alpha parameter for the Zipf distribution (if `skewed` is used). #### 2.6.3 Profiling Tools for deep performance analysis. * `--profile`: Enables Torch Profiler (Requires `SGLANG_TORCH_PROFILER_DIR` env var on server). * `--plot-throughput`: Generates throughput/concurrency plots (requires `termplotlib` and `gnuplot`). * `--profile-activities`: Activities to profile (CPU, GPU, CUDA\_PROFILER). * `--profile-num-steps`: Number of steps to profile. * `--profile-by-stage` / `--profile-stages`: Profile specific processing stages. #### 2.6.4 PD Disaggregation For benchmarking Prefill-Decode (PD) separated architectures. * `--pd-separated`: Enable PD disaggregation benchmarking. * `--profile-prefill-url`: URL(s) of prefill workers for profiling. * `--profile-decode-url`: URL(s) of decode workers for profiling. Note: In PD mode, `prefill` and `decode` must be profiled separately. ### 2.7 Specialized Datasets #### 2.7.1 Generated Shared Prefix (GSP): Designed to test system prompt caching/prefix sharing performance. * `--gsp-num-groups`: Number of unique system prompts. * `--gsp-prompts-per-group`: How many user questions share the same system prompt. * `--gsp-system-prompt-len`: Length of the shared prefix. * `--gsp-fast-prepare`: Skips some statistics calculation for faster startup. #### 2.7.2 Mooncake Designed for trace replay. * `--mooncake-slowdown-factor`: Slows down the trace replay (e.g., 2.0 = 2x slower). * `--mooncake-num-rounds`: Number of conversation rounds (supports multi-turn). * `--use-trace-timestamps`: Schedules requests based on timestamps found in the trace file. ## 3. Metrics After running the benchmark, the tool generally reports: * `E2E` (End-to-End Latency): The total time from sending the request to receiving the final token. * `TTFT` (Time To First Token): The time between sending the request and seeing the first word appear. This represents the Prefill time (processing the image and text prompt). * `TPOT` (Time per Output Token): The average time it takes to generate one token (excluding the first one). This is calculated per request. * `ITL` (Inter-Token Latency): The time gap between two distinct streaming packets. While TPOT is an average, ITL measures the "jitter" or smoothness of the stream. # Diffusion Models Benchmark Documentation Source: https://docs.sglang.io/cookbook/base/benchmarks/diffusion_model_benchmark `sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of diffusion models. It selects the image or video API from the requested task and offers flexible configurations for request rates, dataset types, and profiling. ## 1. Quick Start ### 1.1 Benchmarking in Low Concurrency Run a benchmark on a local server (port 30000) generating 1 videos/images from the `vbench` dataset. ```bash Command theme={null} # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` ### 1.2 Benchmarking in High Concurrency Run a benchmark on a local server (port 30000) generating 20 videos/images from the `vbench` dataset. ```bash Command theme={null} # For text to video: such as Wan2.2-T2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 # For image to video: such as Wan2.2-I2V-A14B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For image-text to video: such as Wan2.2-TI2V-5B-Diffusers python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20 # For text to image: such as Qwen-Image python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 # For image-text to image: such as Qwen-Image-Edit python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` ## 2. Parameter Reference ### 2.1 Connection Settings
Argument Default Description
`--base-url` `None` Base URL of the server (e.g., `http://localhost:30000`). If specified, this overrides `--host` and `--port`.
`--host` `None` The server host (e.g., `127.0.0.1`).
`--port` `None` The server port.
`--model` `None` Model name or path.
### 2.2 Workload & Task Configuration
Argument Choices Description
`--task` `text-to-video`, `image-to-video`, `text-to-image`, `image-to-image`, `video-to-video` Defines the generation task when it cannot be inferred from the model metadata.
`--dataset` `vbench`, `random` The source of prompts/inputs.
`--dataset-path` `None` (Optional) Path to a local dataset file if not using built-in presets.
`--num-prompts` `None` The total number of prompts/requests to execute during the benchmark.
### 2.3 Generation Parameters
Argument Description
`--width` The target width for the generated image or video.
`--height` The target height for the generated image or video.
`--num-frames` Number of frames to generate (Specific to Video backends).
`--fps` Frames Per Second configuration (Specific to Video backends).
### 2.4 Concurrency & Load Control
Argument Description
`--request-rate` The number of requests initiated per second. If set to `inf`, all requests are sent immediately (burst). If set to a number, request arrival times follow a Poisson process.
`--max-concurrency` The maximum number of requests allowed to execute simultaneously. This simulates a semaphore or upstream limit. Even if `request-rate` is high, the actual processing rate is capped by this value.
### 2.5 Logging & Output
Argument Description
`--output-file` Path to save the benchmark metrics (JSON format).
`--disable-tqdm` If set, disables the progress bar in the console.
## 3. Metrics * `Request Throughput` (req/s), Output Throughput (tok/s) * `Latency Mean` (ms): Time to Per Step * `Peak Memory Max` (ms): Max Memory Usage during running # Server Arguments Source: https://docs.sglang.io/cookbook/base/reference/server_arguments This guide explains the parallelism configuration fields used in SGLang model configurations and how they map to SGLang server command-line arguments. ## Quick Reference
Config Field SGLang CLI Argument Description
`tp` `--tp-size`, `--tensor-parallel-size` Tensor Parallelism - splits model across GPUs
`dp` `--dp-size`, `--data-parallel-size` Data Parallelism - runs multiple model replicas
`ep` `--ep-size`, `--expert-parallel-size`, `--ep` Expert Parallelism - distributes MoE experts
`enable_dp_attention` `--enable-dp-attention` DP for attention, TP for FFN (hybrid)
# Cosmos3 Source: https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3 ## 1. Model Introduction [NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler. Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository. | Model | Status | Notes | | ---------------------------------------- | --------- | ------------------------------------------------------------ | | `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V, V2V, joint sound, and action | | `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint | | `nvidia/Cosmos3-Super-Text2Image` | Supported | T2I-specialized checkpoint | | `nvidia/Cosmos3-Super-Image2Video` | Supported | I2V-specialized checkpoint | | `nvidia/Cosmos3-Nano-Policy-DROID` | Supported | DROID policy action generation | | `nvidia/Cosmos3-Edge` | Supported | 4B dense model for T2I, T2V, I2V, V2V, and action generation | | `nvidia/Cosmos3-Edge-Policy-DROID` | Supported | 4B DROID policy action generation | | `nvidia/Cosmos3-Super-Text2Image-4Step` | Supported | 64B T2I checkpoint distilled to a fixed 4-step schedule | | `nvidia/Cosmos3-Super-Image2Video-4Step` | Supported | 64B I2V checkpoint distilled to a fixed 4-step schedule | Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native `FlowUniPCMultistepScheduler`; distilled checkpoints use the fixed sigma schedule stored in the checkpoint. The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all action modes, and `3.0` for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level `flow_shift`. ## 2. Installation Install SGLang with the diffusion dependencies: ```bash Command theme={null} pip install -e "python[diffusion]" ``` Cosmos3 guardrails are enabled by default when the package is available: ```bash Command theme={null} pip install "cosmos-guardrail==0.3.1" ``` `cosmos-guardrail` downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` before starting the server. There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the *\_pickle.UnpicklingError* error occurs during startup, you should change `weight_only=True` to `weights_only=False` parameter in *cosmos\_guardrail/cosmos\_utils.py*: ``` #!/usr/bin/env bash COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")" sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py" ``` ## 3. Serve Cosmos3 Serve `Cosmos3-Nano` directly from the Hugging Face model ID: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Nano \ --num-gpus 1 ``` With `--performance-mode auto`, a Cosmos3 Nano checkpoint keeps its DiT and VAE resident when every selected GPU has at least 120 GiB available at startup. Below that threshold, auto mode retains the conservative DiT component-offload policy. This high-memory override is limited to Nano; Cosmos3 Super checkpoints keep their existing multi-GPU placement defaults. For `Cosmos3-Super`, split the model across multiple GPUs: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Super \ --num-gpus 4 ``` The server also accepts the specialized `nvidia/Cosmos3-Super-Text2Image` and `nvidia/Cosmos3-Super-Image2Video` checkpoint IDs. ### Edge checkpoints `Cosmos3-Edge` is a 4B dense model and can be served on one GPU: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Edge \ --num-gpus 1 ``` Edge is trained for 256p and 480p generation. Its default video configuration is `832x480` with `guidance_scale=5.0`; its default image configuration is `640x640` with `guidance_scale=7.0`. Supported sizes are `832x480`, `480x832`, `640x480`, `480x640`, `480x480`, `640x640`, `448x256`, `256x448`, and `256x256`. Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with `nvidia/Cosmos3-Edge-Policy-DROID`. ### Distilled checkpoints The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU: ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Super-Text2Image-4Step \ --num-gpus 4 ``` For distilled I2V, replace the model path with `nvidia/Cosmos3-Super-Image2Video-4Step`. SGLang detects both checkpoints from `scheduler/scheduler_config.json`, uses the checkpoint's fixed four-step sigma schedule, and forces `guidance_scale=1.0`. Do not tune `num_inference_steps` or `flow_shift` for these checkpoints. ## 4. OpenAI-Compatible Requests ### Text to image Cosmos3 text-to-image uses `/v1/images/generations`. The default Cosmos3 image response is `b64_json`, matching vLLM-Omni's examples. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "A warehouse robot folds a blue cloth on a clean workbench.", "size": "1280x720", "n": 1, "num_inference_steps": 35, "guidance_scale": 6.0, "flow_shift": 3.0, "seed": 0, "extra_args": { "use_resolution_template": false, "guardrails": true } }' ``` With a server running `nvidia/Cosmos3-Super-Text2Image-4Step`, omit the scheduler controls and use `guidance_scale=1.0`: ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "A warehouse robot folds a blue cloth on a clean workbench.", "size": "640x640", "n": 1, "guidance_scale": 1.0, "seed": 0, "extra_args": { "use_resolution_template": false, "guardrails": true } }' ``` ### Text to video with sound Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video. ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \ --form-string "negative_prompt=blurry, distorted, low quality" \ --form-string "size=1280x720" \ --form-string "num_frames=81" \ --form-string "fps=24" \ --form-string "num_inference_steps=35" \ --form-string "guidance_scale=4.0" \ --form-string "flow_shift=10.0" \ --form-string "generate_sound=true" \ --form-string "seed=42" \ --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') while true; do status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \ | python -c 'import json, sys; print(json.load(sys.stdin)["status"])') [ "$status" = "completed" ] && break [ "$status" = "failed" ] && exit 1 sleep 1 done curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \ -o cosmos3_t2v.mp4 ``` ### Image to video This mirrors the official `nvidia/Cosmos3-Nano` Hugging Face image-to-video example: ```python Python theme={null} import json import time from pathlib import Path import requests from huggingface_hub import snapshot_download base_url = "http://127.0.0.1:30010" model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano")) asset_dir = model_dir / "assets" prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text())) negative_prompt = json.dumps( json.loads((asset_dir / "negative_prompt.json").read_text()) ) data = { "prompt": prompt, "negative_prompt": negative_prompt, "size": "1280x720", "num_frames": "189", "fps": "24", "num_inference_steps": "35", "guidance_scale": "6.0", "max_sequence_length": "4096", "flow_shift": "10.0", "seed": "1111", "extra_params": json.dumps( { "use_resolution_template": False, "use_duration_template": False, "guardrails": True, } ), } with (asset_dir / "example_i2v_input.jpg").open("rb") as image: response = requests.post( f"{base_url}/v1/videos", data=data, files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")}, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300) response.raise_for_status() Path("cosmos3_i2v.mp4").write_bytes(response.content) ``` For the distilled I2V checkpoint, use the same API with a server running `nvidia/Cosmos3-Super-Image2Video-4Step`. The recommended request is 480p and does not specify scheduler controls: ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \ --form "input_reference=@first_frame.png;type=image/png" \ --form-string "size=832x480" \ --form-string "num_frames=189" \ --form-string "fps=24" \ --form-string "guidance_scale=1.0" \ --form-string "seed=42" \ --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') ``` Poll and download this job with the same status and content endpoints used by the T2V example. ### Video to video Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source. ```bash Command theme={null} job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \ --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \ --form "video_reference=@robot_pouring.mp4;type=video/mp4" \ --form-string "size=1280x704" \ --form-string "num_frames=45" \ --form-string "fps=24" \ --form-string "num_inference_steps=35" \ --form-string "guidance_scale=6.0" \ --form-string 'condition_frame_indexes=[0,1]' \ --form-string "condition_video_keep=first" \ | python -c 'import json, sys; print(json.load(sys.stdin)["id"])') ``` Poll and download this job with the same status and content endpoints used by the T2V example. ### Action generation For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism. ```bash Command theme={null} sglang serve \ --model-path nvidia/Cosmos3-Nano-Policy-DROID \ --num-gpus 1 ``` Use `nvidia/Cosmos3-Edge-Policy-DROID` in the same command to serve the smaller 4B policy checkpoint. `policy` and `inverse_dynamics` return actions, so their canonical API is the synchronous `/v1/actions/generations` endpoint. The following request predicts a 16-step action chunk from one observation image. `action_horizon=16` maps to the model's `num_frames=17` convention. ```python Python theme={null} import base64 from pathlib import Path import requests image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode() response = requests.post( "http://127.0.0.1:30010/v1/actions/generations", json={ "input": { "task": "Put the pot to the left of the purple item.", "observation": { "image": {"b64_json": image_b64}, }, }, "parameters": { "action_mode": "policy", "action_horizon": 16, "domain_name": "droid_lerobot", "height": 480, "width": 832, "fps": 5, "num_inference_steps": 30, "guidance_scale": 1.0, "seed": 42, }, }, timeout=300, ) response.raise_for_status() action = response.json()["data"][0]["action"] print(action["shape"], action["values"]) ``` Use `GET /v1/actions/metadata` to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the `/v1/actions/realtime` websocket use the same action envelope. `inverse_dynamics` also uses `/v1/actions/generations`; set `action_mode="inverse_dynamics"` and pass an observation video URL or server-local path as `input.observation.video`. Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name. `forward_dynamics` is intentionally different: it consumes an action array and predicts video, so it remains on `/v1/videos`. Action-producing modes submitted to `/v1/videos` return HTTP 400 with the canonical action endpoint in the error message. ## 5. Cosmos3 Parameters Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`. For distilled checkpoints, SGLang replaces `num_inference_steps` with the checkpoint's fixed four-step schedule and forces `guidance_scale=1.0`; negative-prompt CFG and request-level `flow_shift` do not apply. Top-level Cosmos3 request fields: * `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer. * `flow_shift`: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I, `10.0` for non-Edge video and all action modes, or `3.0` for Edge video). * `guidance_interval`: optional `[start, end]` noise interval for CFG. Non-distilled T2I defaults to `[400, 1000]`; video modes guide at every step. Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields: * `generate_sound`: generate a sound track whose duration follows `num_frames / fps`. * `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration. * `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`. * `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning. * `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`. * `domain_name` / `domain_id`: select the action embodiment head. * `raw_action_dim`: number of active action dimensions; inferred for known domain names. * `action`: action array with shape `[T, D]`, required by `forward_dynamics`. * `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS. * `action_view_point`: viewpoint used in the structured action caption. * `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`. Put model-specific compatibility knobs in `extra_params` for video requests, or `extra_args` for image requests: * `use_duration_template`: whether to append SGLang's generated duration suffix to video prompts. * `use_resolution_template`: accepted for vLLM-Omni request compatibility. * `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template. * `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled. # ERNIE-Image Source: https://docs.sglang.io/cookbook/diffusion/Ernie-Image/Ernie-Image ## 1. Model Introduction [ERNIE-Image](https://huggingface.co/baidu/ERNIE-Image) is Baidu's text-to-image family with separate standard and Turbo checkpoints. The standard model is the quality-oriented path; Turbo is the lower-latency choice, and both load through SGLang's native `ErnieImagePipeline`. This integration currently targets text-only image generation rather than image editing or reference conditioning. Choose it for a straightforward single-GPU T2I deployment; use a dedicated editing model when preserving source-image structure is part of the task. | Model | Hugging Face model ID | Notes | | ----------------- | ------------------------- | -------------------------------- | | ERNIE-Image | `baidu/ERNIE-Image` | Regular text-to-image checkpoint | | ERNIE-Image-Turbo | `baidu/ERNIE-Image-Turbo` | Turbo text-to-image checkpoint | ## 2. Installation Install SGLang with the diffusion dependencies: ```bash Command theme={null} pip install -e "python[diffusion]" ``` For full installation options, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 3. Serve the model The commands below target a single supported NVIDIA CUDA or AMD ROCm GPU. Start with `--performance-mode auto`; use `speed` only when the full pipeline fits comfortably on the selected GPU(s), and use `memory` when you need lower peak GPU memory. Serve ERNIE-Image: ```bash Command theme={null} sglang serve \ --model-path baidu/ERNIE-Image \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Serve ERNIE-Image-Turbo: ```bash Command theme={null} sglang serve \ --model-path baidu/ERNIE-Image-Turbo \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` ## 4. Generate an image Use the OpenAI-compatible image generation API after the server starts: ```python Python theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1") response = client.images.generate( model="baidu/ERNIE-Image-Turbo", prompt="A cinematic photo of a quiet lakeside cabin at sunrise", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("ernie_image.png", "wb") as f: f.write(image_bytes) ``` ## 5. Configuration tips * ERNIE-Image is a text-to-image pipeline; do not pass `--image-path`. * `--performance-mode auto` keeps conservative defaults while preserving explicit user flags. * If the checkpoint includes a PE component, SGLang loads it automatically with the native Ministral3 runtime. Use `--layerwise-offload-components pe` when the local PE decoder needs to trade latency for lower GPU memory usage. * Treat FSDP, SP/Ulysses/Ring, and TP as explicit benchmark knobs. Measure the target resolution, step count, and GPU type before making them production defaults. # FLUX Source: https://docs.sglang.io/cookbook/diffusion/FLUX/FLUX ## 1. Model Introduction [FLUX](https://blackforestlabs.ai/) is Black Forest Labs' rectified-flow image model family. [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is the smaller 12B text-to-image checkpoint; [FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32B model that adds instruction-based editing plus single- and multi-reference composition. FLUX is a strong default when prompt adherence, polished image quality, or reference consistency matters. The tradeoff is deployment weight: FLUX.2 needs substantially more memory than FLUX.1, and the dev checkpoints use the FLUX non-commercial license, so review the model license before production use. | Checkpoint | Best fit | Main limitation | | ------------------------------ | --------------------------------------------------------------------- | ---------------------------------------- | | `black-forest-labs/FLUX.1-dev` | High-quality text-to-image with the lighter FLUX deployment | No native multi-reference editing path | | `black-forest-labs/FLUX.2-dev` | Text-to-image, editing, and reference-guided composition in one model | 32B model with a larger memory footprint | ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2, A3 NPUs. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1") response = client.images.generate( model="black-forest-labs/FLUX.1-dev", prompt="A cat holding a sign that says hello world", size="1024x1024", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path black-forest-labs/FLUX.1-dev ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path black-forest-labs/FLUX.1-dev ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark ### 5.1 Speedup Benchmark #### 5.1.1 Generate a image Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: black-forest-labs/FLUX.1-dev * sglang diffusion version: 0.5.6.post2 **Server Command**: ```shell Command theme={null} sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: black-forest-labs/FLUX.1-dev Dataset: vbench Task: text-to-image -------------------------------------------------- Benchmark duration (s): 50.97 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.02 Latency Mean (s): 50.9681 Latency Median (s): 50.9681 Latency P99 (s): 50.9681 -------------------------------------------------- Peak Memory Max (MB): 27905.19 Peak Memory Mean (MB): 27905.19 Peak Memory Median (MB): 27905.19 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench -------------------------------------------------- Benchmark duration (s): 16.30 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.06 Output throughput (outputs/s): 0.06 Latency Mean (s): 16.30 Latency Median (s): 16.30 Latency P90 (s): 16.30 Latency P95 (s): 16.30 Latency P99 (s): 16.30 -------------------------------------------------- Peak Memory Max (MB): 19972.00 Peak Memory Mean (MB): 19972.00 Peak Memory Median (MB): 19972.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Server Command** : ```shell Command theme={null} sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000 ``` **Benchmark Command** : ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result** : ```text Output theme={null} ================= Serving Benchmark Result ================= Model: black-forest-labs/FLUX.1-dev Dataset: vbench Task: text-to-image -------------------------------------------------- Benchmark duration (s): 111.79 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.18 Latency Mean (s): 67.0646 Latency Median (s): 66.9691 Latency P99 (s): 110.8949 -------------------------------------------------- Peak Memory Max (MB): 27917.19 Peak Memory Mean (MB): 27916.59 Peak Memory Median (MB): 27917.19 ============================================================ ``` **Server Command** : ```shell Command theme={null} #One A3 card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` **Benchmark Command** : ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result** : ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: black-forest-labs/FLUX.1-dev Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.85 Request rate: inf Max request concurrency: 20 Successful requests: 18/20 Completed outputs: 18 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.06 Output throughput (outputs/s): 0.06 Latency Mean (s): 155.16 Latency Median (s): 155.11 Latency P90 (s): 266.30 Latency P95 (s): 280.15 Latency P99 (s): 291.23 -------------------------------------------------- Peak Memory Max (MB): 19972.00 Peak Memory Mean (MB): 19972.00 Peak Memory Median (MB): 19972.00 ------------------------------------------------------------ ``` # Ideogram 4 Source: https://docs.sglang.io/cookbook/diffusion/Ideogram/Ideogram4 ## 1. Model Introduction [Ideogram 4](https://huggingface.co/ideogram-ai/ideogram-4-nf4) is a text-to-image family focused on polished composition and typography-aware generation. SGLang supports the official NF4 and FP8 checkpoints, Comfy-Org's NVFP4 transformer, and fal's distilled Fast and Instant variants. Choose NF4 when memory is the primary constraint, FP8/NVFP4 for supported accelerator deployments, Fast for a 20-step balance, and Instant for the 8-step latency target. The public repositories are gated, and the available releases are already quantized or distilled rather than a single full-precision baseline. | Variant | Hugging Face model ID | Notes | | ------- | ---------------------------- | --------------------------------------------------------------------------------------------------------------- | | NF4 | `ideogram-ai/ideogram-4-nf4` | Official bitsandbytes NF4 checkpoint. Use this path first for low-memory deployment. | | FP8 | `ideogram-ai/ideogram-4-fp8` | Official FP8 checkpoint. | | NVFP4 | `Comfy-Org/Ideogram-4` | Comfy-Org NVFP4 transformer weights. SGLang loads non-transformer components from `ideogram-ai/ideogram-4-fp8`. | | Fast | `fal/ideogram-v4-fast` | 20-step, CFG-distilled, FP4-targeted floating checkpoint. Defaults to `V4_FAST_20`. | | Instant | `fal/ideogram-v4-instant` | 8-step, CFG- and timestep-distilled BF16 checkpoint. Defaults to `V4_INSTANT_8`. | The fal repositories contain only the transformer component. When either model ID is passed directly to `--model-path`, SGLang loads the text encoder, tokenizer, VAE, and scheduler from the `ideogram-ai/ideogram-4-nf4-diffusers` revision referenced by fal's model cards, then runs the distilled transformer without the unconditional branch. ## 2. Prerequisites * NVIDIA CUDA GPU. * SGLang installed with diffusion dependencies. * `bitsandbytes>=0.46.1` and `accelerate>=1.1.0` for the NF4 checkpoint and the fal variants' shared NF4 text encoder. * `HF_TOKEN` with access to the Ideogram 4 gated repositories. ## 3. Serve the model NF4: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path ideogram-ai/ideogram-4-nf4 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` FP8: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path ideogram-ai/ideogram-4-fp8 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Comfy-Org NVFP4: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path Comfy-Org/Ideogram-4 \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` Use B200 or another Blackwell GPU for NVFP4. fal Fast: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-fast \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` fal Instant: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 1 \ --performance-mode auto \ --port 30010 ``` ### Tensor and sequence parallelism Both fal variants support native DiT tensor parallelism and Ulysses sequence parallelism. The following two-GPU layouts were validated at 1024×1024 on B200 GPUs for both `fal/ideogram-v4-fast` and `fal/ideogram-v4-instant`. TP2 shards the distilled DiT weights. The shared bitsandbytes NF4 text encoder is replicated because 4-bit row-parallel quantization states cannot be safely sharded: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 2 \ --tp-size 2 \ --port 30010 ``` Ulysses/SP2 shards the image-token sequence while keeping the DiT weights replicated: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 2 \ --ulysses-degree 2 \ --port 30010 ``` Replace the model ID with `fal/ideogram-v4-fast` to use the same layouts for Fast. TP2 and Ulysses/SP2 are the validated configurations; Ring SP is not yet validated for this pipeline. Ideogram 4 has 18 attention heads, so any other TP degree must divide 18. ### Layerwise offload If the distilled DiT does not fit in GPU memory, enable transformer layerwise offload. This keeps the shared NF4 text encoder on the GPU while streaming the DiT blocks from pinned CPU memory. It reduces peak VRAM at the cost of additional latency: ```bash Command theme={null} HF_TOKEN=$HF_TOKEN sglang serve \ --model-path fal/ideogram-v4-instant \ --num-gpus 1 \ --dit-layerwise-offload \ --layerwise-offload-components transformer \ --port 30010 ``` The same option applies to Fast. Layerwise offload was validated with a real Instant generation; TP2 and Ulysses/SP2 were validated separately without offload. The released Fast weights are stored in a floating-point pre-pack form and can be loaded directly, but fal trained them with QAD for an NVFP4 execution path. SGLang does not currently quantize dense NVIDIA DiTs to NVFP4 at load time, so direct Fast inference bypasses the intended quantization path and may be visibly degraded. Treat direct floating-point Fast inference as compatibility/testing support, not the production-quality path. The released Instant checkpoint is BF16 and is the recommended local checkpoint today. ## 4. Generate an image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1") response = client.images.generate( model="ideogram-ai/ideogram-4-nf4", prompt="A cinematic poster of a quiet bookstore at dusk with elegant hand-lettered signage", size="1024x1024", n=1, response_format="b64_json", extra_body={"preset": "V4_QUALITY_48", "seed": 0}, ) image_bytes = base64.b64decode(response.data[0].b64_json) with open("ideogram4.png", "wb") as f: f.write(image_bytes) ``` fal's hosted endpoint expands natural-language prompts, but these local checkpoints expect Ideogram 4's structured JSON caption format. Pass a complete caption as the API `prompt`, for example: ```python Example theme={null} import json prompt = json.dumps( { "high_level_description": ( "A bold typographic poster centered on the exact words INSTANT BY FAL, " "printed in black and electric orange on warm white paper." ), "compositional_deconstruction": { "background": "Warm white textured paper with generous negative space.", "elements": [ { "type": "text", "text": "INSTANT BY FAL", "desc": "Large uppercase geometric sans-serif lettering, precisely centered.", } ], }, }, ensure_ascii=False, separators=(",", ":"), ) response = client.images.generate( model="fal/ideogram-v4-instant", prompt=prompt, size="1024x1024", n=1, response_format="b64_json", extra_body={"seed": 42}, ) ``` Base Ideogram 4 presets are `V4_DEFAULT_20`, `V4_QUALITY_48`, and `V4_TURBO_12`. The fal variants automatically select `V4_FAST_20` and `V4_INSTANT_8`, respectively. A preset controls both `num_inference_steps` and guidance, so do not set those fields directly. # JoyAI-Echo Source: https://docs.sglang.io/cookbook/diffusion/JoyEcho/JoyEcho Run JoyAI-Echo multi-shot audio–video generation with SGLang Diffusion. ## 1. Model Introduction [JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) is an 8-step long-form audio-video model built on LTX-2. Its paired memory bank carries decoded visual context and audio latents across prompt changes, making it strongest for multi-shot, minute-scale sequences that need continuity in both picture and soundtrack. Choose JoyEcho over a standard LTX pipeline when shots must share audiovisual memory. Its distilled 832×480 path prioritizes long-form continuity and throughput rather than the higher-resolution two-stage quality modes offered by LTX-2.3. Use `jdopensource/JoyAI-Echo` as `--model-path`; SGLang materializes the monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). | Aspect | Standard LTX-2.3 | JoyEcho | | -------------------- | ----------------------------------------- | ------------------------------------------------------------------------- | | Pipeline | `LTX2Pipeline` / `LTX2TwoStageHQPipeline` | `JoyEchoPipeline` (default for this model) | | Denoising | Multi-step flow matching + CFG | LTX-2 DMD distilled path (8 steps, `guidance_scale=1.0`) | | Multi-shot | Not supported | Paired audio–video memory bank across shots | | Sequence parallelism | LTX-2 SP (video/audio sharded) | Ulysses SP (`ulysses_degree=2`): single-shot and multi-shot + memory bank | | Post-processing | Optional two-stage HQ upscaling | Per-shot mp4 output | Review the model license on the [JoyAI-Echo Hugging Face page](https://huggingface.co/jdopensource/JoyAI-Echo) before production or commercial use. SGLang support does not grant additional model usage rights. ## 2. SGLang-diffusion Installation Install SGLang with diffusion dependencies: ```bash theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` For platform-specific setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 3. Model Deployment JoyEcho uses the default `JoyEchoPipeline` registered for `jdopensource/JoyAI-Echo`. A single high-VRAM GPU (for example H100 or H200) is enough for the common 832x480 / 121-frame / 8-step setting. ```bash theme={null} sglang serve \ --model-path jdopensource/JoyAI-Echo ``` Optional environment variable for long runs: ```bash theme={null} export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True ``` For multi-GPU serving, tensor parallelism (TP) and **Ulysses sequence parallelism (SP)** are supported. JoyEcho SP uses an **asymmetric layout**: video target latents are time-sharded across ranks, while audio (including memory tokens) is **replicated** on every rank so cross-attention stays temporally aligned. Multi-shot runs with `enable_memory_bank=true` are supported on SP. ```bash theme={null} sglang serve \ --model-path jdopensource/JoyAI-Echo \ --num-gpus 2 \ --ulysses-degree 2 ``` JoyEcho SP currently targets **Ulysses-only** parallelism (`ulysses_degree=2`, `ring_degree=1`). Ring SP is not validated for this pipeline. For `sglang generate`, add `--num-gpus 2 --ulysses-degree 2` to the commands in section 4. ## 4. Model Invocation ### 4.1 Default sampling | Setting | Default | | -------------- | ------- | | Resolution | 832x480 | | Frames | 121 | | FPS | 25 | | Steps | 8 | | Guidance scale | 1.0 | | Seed | 12345 | ### 4.2 Single-shot text-to-video ```bash theme={null} sglang generate \ --model-path jdopensource/JoyAI-Echo \ --prompt "A curious raccoon walks through a sunlit forest path" \ --height 480 --width 832 --num-frames 121 --fps 25 \ --num-inference-steps 8 --seed 42 \ --save-output ``` Disable the memory bank for standalone clips with a config file: ```bash theme={null} cat > /tmp/joy_echo_single.json <<'EOF' { "model_path": "jdopensource/JoyAI-Echo", "prompt": "A curious raccoon walks through a sunlit forest path", "enable_memory_bank": false, "seed": 42, "height": 480, "width": 832, "num_frames": 121, "fps": 25, "num_inference_steps": 8 } EOF sglang generate --config /tmp/joy_echo_single.json --save-output ``` ### 4.3 Multi-shot generation JoyEcho does **not** generate all shots in one forward pass. Each shot is one generation request. Continuity is carried by an in-process **memory bank** on the pipeline instance. Typical workflow: 1. **Shot 0** — memory bank is empty; the model generates a standalone A/V clip. 2. **After decode** — decoded video frames and packed audio latents are committed to the memory bank (up to 7 slots by default). 3. **Shot 1+** — prior-shot frames are re-encoded and prepended as a memory prefix before denoising. 4. **Per-shot seeding** — official semantics use `prompt_seed = base_seed + shot_index`. Pass multiple prompts as a list in a config file: ```bash theme={null} cat > /tmp/joy_echo_4shot.json <<'EOF' { "model_path": "jdopensource/JoyAI-Echo", "prompt": [ "Shot 0: A raccoon wakes up in a cozy attic.", "Shot 1: The raccoon climbs down and opens the back door.", "Shot 2: It walks through a rainy alley under neon signs.", "Shot 3: The raccoon finds a warm bakery window and stops." ], "enable_memory_bank": true, "reset_memory_bank": true, "seed": 42, "height": 480, "width": 832, "num_frames": 121, "fps": 25, "num_inference_steps": 8 } EOF sglang generate --config /tmp/joy_echo_4shot.json --save-output ``` You can also pass prompts from a text file (one prompt per line) with `--prompt-path`: ```bash theme={null} sglang generate \ --model-path jdopensource/JoyAI-Echo \ --prompt-path /tmp/joy_echo_shots.txt \ --seed 42 \ --height 480 --width 832 --num-frames 121 --fps 25 \ --num-inference-steps 8 \ --save-output ``` ### 4.4 Memory bank controls | Parameter | Default | Meaning | | -------------------- | ------- | -------------------------------------------------------------------------------------------------- | | `enable_memory_bank` | `true` | Read/write the paired A/V memory bank between shots. | | `reset_memory_bank` | `true` | Clear the bank and shot counter at the start of a new session (`request_id` change or first shot). | Set `enable_memory_bank=false` when you want independent shots without cross-shot continuity. ## 5. Practical Tips * Use `--num-inference-steps 8` and `--guidance-scale 1.0` to match the official JoyEcho DMD distilled path. * Multi-shot prompts can be passed as a `prompt` list, via `prompt_path`, or as sequential API calls on the same server instance. * The memory bank caps at **7 slots**; from shot 8 onward the oldest slots roll off. * For **2-GPU latency**, try **Ulysses SP** (`--num-gpus 2 --ulysses-degree 2`) on both single-shot and multi-shot runs. Use **TP** when you need a different sharding strategy or more than two GPUs. * Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for long multi-shot SP sessions. * JoyEcho outputs per-shot mp4 files with synchronized audio. There is no built-in two-stage HQ upscaling path like LTX-2.3 HQ. # Krea-2 Source: https://docs.sglang.io/cookbook/diffusion/Krea/Krea-2 ## 1. Model Introduction [Krea-2](https://huggingface.co/krea/Krea-2-Turbo) is Krea's photorealistic text-to-image family, built as a single-stream MMDiT with a Qwen3-VL text encoder and Qwen-Image VAE. Both public variants use the same native SGLang pipeline and differ mainly in their sampling target. Choose Turbo for interactive generation: it is distilled to 8 steps with `guidance_scale=1.0`. Choose Raw when maximum fidelity matters more than latency: it uses roughly 52 steps with classifier-free guidance. Neither checkpoint is an image-editing model; use the Qwen-Image-Edit or FLUX.2 path when an input image must be preserved or transformed. | Variant | Model ID | Sampling profile | | ------- | ------------------- | --------------------------------------------- | | Turbo | `krea/Krea-2-Turbo` | 8 steps, no CFG; fastest path | | Raw | `krea/Krea-2-Raw` | About 52 steps with CFG; higher-fidelity path | ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section covers deploying Krea-2-Turbo for fast, high-quality image generation. ### 3.1 Basic Configuration Krea-2-Turbo generates high-quality images in only 8 inference steps. Launch the server with: ```bash Command theme={null} sglang serve \ --model-path krea/Krea-2-Turbo \ --num-gpus 1 \ --port 30000 ``` The step count and guidance scale are **request-time** settings (see [API Usage](#4-api-usage)); Krea-2-Turbo defaults to 8 steps with `guidance_scale = 1.0`. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--num-gpus`: Number of GPUs to use. * Multi-GPU (tensor and/or sequence parallelism): see [Section 3.3](#3-3-multi-gpu-tensor-and-sequence-parallelism). ### 3.3 Multi-GPU: tensor and sequence parallelism Krea-2 supports two multi-GPU axes that can be combined; `--num-gpus` must equal `tp_size × ulysses_degree`. * **Tensor parallelism (`--tp-size N`)** shards the DiT weights across GPUs, lowering per-GPU VRAM. Krea-2's attention heads (48 query / 12 KV) and text heads (20) are divisible by a tp size of 1, 2, or 4. * **Sequence parallelism / Ulysses (`--ulysses-degree N`)** shards the image-token sequence across GPUs while keeping the text prefix replicated. It does **not** shard weights (per-GPU VRAM is unchanged), but its output is **bitwise-identical** to single-GPU. It currently requires a single prompt per request (ragged/padded multi-prompt batches under SP are not supported — use `--tp-size` for those). ```bash Command theme={null} # Tensor parallel (2 GPUs) — lowest per-GPU VRAM (DiT weights sharded) sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --tp-size 2 --port 30000 # Sequence parallel / Ulysses (2 GPUs) — output bitwise-identical to single-GPU sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --ulysses-degree 2 --port 30000 # Hybrid TP × SP (4 GPUs) — composes both axes sglang serve --model-path krea/Krea-2-Turbo --num-gpus 4 --tp-size 2 --ulysses-degree 2 --port 30000 ``` Measured on 2× H200 (Krea-2-Turbo, 8 steps, 1024×1024): `--tp-size 2` and `--ulysses-degree 2` each give \~1.7× denoise speedup over single-GPU; the hybrid TP=2 × SP=2 reaches \~2.8× on 4 GPUs. **Choosing:** on memory-constrained GPUs prefer `--tp-size` (it shards the \~24 GB DiT, e.g. \~38 GB → \~27 GB per GPU on 2 GPUs); on large-VRAM GPUs sequence parallelism is marginally faster and numerically exact, and the two compose for the highest throughput. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image Generate an image with the OpenAI-compatible images API: ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="krea/Krea-2-Turbo", prompt="a red fox sitting in fresh snow, golden hour, photorealistic", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` You can also generate a single image from the command line: ```bash Command theme={null} sglang generate --model-path krea/Krea-2-Turbo \ --prompt "a red fox sitting in fresh snow, golden hour, photorealistic" \ --num-inference-steps 8 --height 1024 --width 1024 --save-output ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to speed up inference with minimal quality loss. Enable it by setting `SGLANG_CACHE_DIT_ENABLED=true`. For more details, see the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). Cache-DiT works for **both** Krea-2 variants with no extra configuration: SGLang tracks each request's classifier-free-guidance mode, so Krea-2-Turbo (no CFG, `guidance_scale = 1.0`) and Krea-2-Raw (CFG, `guidance_scale ≈ 4.5`) both cache correctly and automatically. **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path krea/Krea-2-Turbo \ --num-gpus 1 \ --port 30000 ``` Measured per-image denoise speedup with the default cache settings (NVIDIA H200, 1024x1024, seed 0): | Variant | Inference steps | Denoise (no cache → cache) | Speedup | | :-------------------- | :-------------- | :------------------------- | :------ | | Krea-2-Turbo (no CFG) | 8 | 1.27s → 0.92s | \~1.4x | | Krea-2-Raw (CFG 4.5) | 50 | 18.0s → 6.3s | \~2.9x | Caching has the most headroom on Raw's longer schedule; the 8-step distilled Turbo has only a few cacheable steps after warmup. **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion (best suited to the longer Raw schedule; not recommended for the 8-step Turbo):
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example (Krea-2-Raw, default cache settings shown explicitly): ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=1 \ SGLANG_CACHE_DIT_BN=0 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.24 \ SGLANG_CACHE_DIT_MC=3 \ sglang serve --model-path krea/Krea-2-Raw ``` #### 4.2.2 Memory and Component Residency Krea-2's DiT is \~24 GB in bf16 (the bulk of the model). On memory-constrained GPUs you can keep less of it resident: * `--component-residency dit=layerwise-offload`: stream the DiT's transformer blocks layer-by-layer with async host-to-device prefetch overlap, so only a small working set stays on the GPU. This is the primary way to fit Krea-2 on a single consumer / 32 GB-class card, at a modest latency cost. Tune the memory/latency trade-off with `--dit-offload-prefetch-size` (`0.0` prefetches one layer for the lowest memory; larger values prefetch more layers -- faster but more memory). * `--component-residency dit=component-offload`: keep the complete DiT on CPU between denoising uses. This and layerwise offload are distinct modes; do not combine them for the same component. * `--component-residency text_encoder=component-offload`: offload the Qwen3-VL text encoder while it is idle during denoising. * `--component-residency vae=component-offload`: offload the VAE between uses. * `--pin-cpu-memory`: pin host memory for offload. Add only as a temporary workaround if you hit `CUDA error: invalid argument`. The legacy `--dit-layerwise-offload`, `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--vae-cpu-offload` forms remain accepted. If both legacy DiT offload flags are enabled, layerwise offload is the effective DiT mode. On large-VRAM GPUs (e.g. H200), keep everything resident (offloads off) for the fastest latency. ## 5. Benchmark Test Environment: * Hardware: NVIDIA H200 GPU (1x) * Model: krea/Krea-2-Turbo (8 inference steps) * sglang diffusion version: 0.5.13 **Server Command** (used for both benchmarks below): ```shell Command theme={null} sglang serve --model-path krea/Krea-2-Turbo --port 30000 ``` ### 5.1 Generate an image **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \ --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: krea/Krea-2-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1.56 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.64 Latency Mean (s): 1.5600 Latency Median (s): 1.5600 Latency P99 (s): 1.5600 -------------------------------------------------- Peak Memory Max (MB): 37466.00 Peak Memory Mean (MB): 37466.00 Peak Memory Median (MB): 37466.00 ============================================================ ``` ### 5.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \ --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: krea/Krea-2-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 31.47 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.64 Latency Mean (s): 16.5000 Latency Median (s): 16.5200 Latency P99 (s): 31.1300 -------------------------------------------------- Peak Memory Max (MB): 37468.00 Peak Memory Mean (MB): 37466.40 Peak Memory Median (MB): 37466.00 ============================================================ ``` # LTX2 & LTX2.3 Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2 & LTX2.3 Run LTX-2 and LTX-2.3 video generation pipelines with SGLang Diffusion. ## 1. Model Introduction [LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are Lightricks video models for text-to-video and image-conditioned generation. Their defining deployment choice is the pipeline: one-stage minimizes latency, standard two-stage refines quality, and LTX-2.3 adds a 1920×1088 HQ path. Choose LTX when one model family must span fast generation, higher-quality refinement, and LoRA adaptation. Two-stage quality costs substantially more memory and startup work because both DiTs and the spatial upsampler participate; use the one-stage path when latency or capacity matters more than final detail. Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. SGLang loads the two-stage upsampler and distilled components from the selected snapshot. **License notice:** LTX-2 and LTX-2.3 are released under the LTX-2 Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://huggingface.co/Lightricks/LTX-2.3/blob/main/LICENSE) before production or commercial use; SGLang support does not grant additional model usage rights. ## 2. SGLang-diffusion Installation Install SGLang with diffusion dependencies: ```bash theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` For platform-specific setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 3. Model Deployment This section provides deployment configurations optimized for different LTX pipelines and hardware targets. ### 3.1 Basic Configuration The LTX series supports one-stage and two-stage pipelines. LTX-2.3 also supports the HQ two-stage pipeline. The recommended launch configuration depends on whether the target GPU can keep both two-stage DiTs resident. **Interactive Command Generator**: Use the configuration selector below to generate a deployment command. The default selection targets a single NVIDIA H200 with `resident` two-stage mode. For multi-GPU serving, start from the 2-GPU or 4-GPU presets and only change parallelism if you need more memory headroom. ### 3.2 Configuration Tips Choose the pipeline class based on the quality and latency target: | Use case | Pipeline class | Notes | | -------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------ | | One-stage generation | `LTX2Pipeline` | Fastest LTX native path. Supports T2V and TI2V. | | Two-stage generation | `LTX2TwoStagePipeline` | Uses a base stage and a refinement stage. Supported by LTX-2 and LTX-2.3. | | Two-stage High Quality (HQ) generation | `LTX2TwoStageHQPipeline` | LTX-2.3 HQ path; defaults to 1920x1088 unless you override `--width` and `--height`. | Feature compatibility: | Pipeline class | T2V | TI2V (`--image-path`) | LoRA (`--lora-path`) | Notes | | ------------------------ | --- | --------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------- | | `LTX2Pipeline` | Yes | Yes | Yes | One-stage path. Cannot be combined with HQ because HQ is a separate two-stage pipeline class. | | `LTX2TwoStagePipeline` | Yes | Yes | Yes | Standard two-stage path for LTX-2 and LTX-2.3. | | `LTX2TwoStageHQPipeline` | Yes | Yes | Yes | High Quality two-stage path for LTX-2.3. Use this instead of `LTX2Pipeline`; it is not a one-stage mode flag. | For two-stage pipelines, `--ltx2-two-stage-device-mode` controls transformer residency: | Mode | When to use it | | ---------- | ------------------------------------------------------------------- | | `resident` | Best latency on high-VRAM GPUs because both DiTs can stay resident. | | `original` | Closest to the original two-stage switching semantics. | Other deployment flags: * `--lora-path`: Preload a community LoRA adapter. * `--lora-weight-name`: Select the exact safetensors file when the LoRA repository contains multiple weight files. For native LTX-2.3 two-stage serving without a user LoRA, `resident` is the fastest high-VRAM path. LTX-2 still applies the distilled LoRA during the stage switch, so `--ltx2-two-stage-device-mode` is mainly an LTX-2.3 optimization. When you pass `--lora-path`, SGLang still applies the user LoRA during the two-stage switch, so use `resident` on H200-class GPUs for enough VRAM, but do not expect the same premerged-stage2 benefit as the no-user-LoRA path. ### 3.3 Fast multi-GPU presets For latency-oriented LTX serving, prefer CFG parallel over sequence parallelism. CFG parallel splits guidance branches across GPUs, while SP/Ulysses is mainly a memory/long-sequence tool for LTX. | Target | Recommended server flags | Notes | | ------------------------ | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | LTX-2.3, 1 high-VRAM GPU | `--ltx2-two-stage-device-mode resident` | Fastest two-stage setup when both DiTs fit. | | LTX-2.3, 1 standard GPU | `--ltx2-two-stage-device-mode original` | Lower VRAM than `resident`; use this when H100-class memory is tight. | | LTX-2, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel` | Fastest verified 2-GPU setup; keep `--dit-layerwise-offload` disabled unless memory is tight. | | LTX-2.3, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 2-GPU setup. | | LTX-2.3, 4 GPUs | `--num-gpus 4 --tp-size 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 4-GPU layout: TP2 inside each CFG branch. | | Official comparison | `--ltx2-two-stage-device-mode original` | Use this only when matching the original LTX-2.3 stage-switch semantics matters. | Use `--enable-cfg-parallel` for degree-2 CFG parallel. Use `--cfg-parallel-size` only when you explicitly need a different CFG branch count. If `resident` exceeds available VRAM, keep the same parallelism preset and switch only the device mode to `original`. On high-VRAM GPUs, add `--text-encoder-cpu-offload false` if text encoding latency matters and you have enough memory. #### 3.3.1 Two GPUs ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --num-gpus 2 \ --enable-cfg-parallel \ --ltx2-two-stage-device-mode resident ``` #### 3.3.2 Four GPUs ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --num-gpus 4 \ --tp-size 2 \ --enable-cfg-parallel \ --ltx2-two-stage-device-mode resident ``` ## 4. Model Invocation ### 4.1 Basic Usage The examples below spell out the current SGLang sampling defaults for reproducibility: | Model path | Default output | Default frames | Default steps | | -------------------------------------------------- | -------------- | -------------- | ------------- | | `Lightricks/LTX-2` | 768x512 | 121 | 40 | | `Lightricks/LTX-2.3` | 768x512 | 121 | 30 | | `Lightricks/LTX-2.3` with `LTX2TwoStageHQPipeline` | 1920x1088 | 121 | 15 | #### 4.1.1 LTX-2 one-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2 \ --pipeline-class-name LTX2Pipeline \ --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \ --save-output ``` #### 4.1.2 LTX-2.3 one-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2Pipeline \ --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \ --save-output ``` #### 4.1.3 LTX-2 two-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2 \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \ --save-output ``` #### 4.1.4 LTX-2.3 two-stage text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \ --save-output ``` #### 4.1.5 LTX-2.3 HQ text-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStageHQPipeline \ --prompt "A wide cinematic shot of alpine clouds rolling over a mountain ridge, soft morning light, slow aerial camera movement" \ --save-output ``` #### 4.1.6 Image-to-video with one reference image Pass one image to `--image-path` for image-conditioned generation: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png \ --prompt "The camera slowly pushes forward as the subject turns toward warm window light, subtle natural motion, cinematic" \ --save-output ``` #### 4.1.7 First-to-last-frame transition with two reference images Pass two images to `--image-path` for transition-style TI2V. The first image is used as the starting condition and the second image is used as the ending condition. ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png ./inputs/end.png \ --prompt "A smooth cinematic transition from the first scene into the final scene, dynamic camera motion, motion blur, zhuanchang" \ --save-output ``` ### 4.2 Advanced Usage #### 4.2.1 Use community LoRAs Use `--lora-path` to load a LoRA adapter. If the Hugging Face repo contains multiple safetensors files, use `--lora-weight-name` to select the exact file. `--lora-scale` maps to the standard LoRA merge scale and defaults to `1.0`. The following example uses [`valiantcat/LTX-2.3-Transition-LORA`](https://huggingface.co/valiantcat/LTX-2.3-Transition-LORA): ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --lora-path valiantcat/LTX-2.3-Transition-LORA \ --lora-weight-name ltx2.3-transition.safetensors \ --prompt "A low-angle tracking shot moves through a foggy forest road. The camera rises above the treetops and transitions into a clear view of a snowy mountain peak under bright sunlight, zhuanchang" \ --save-output ``` You can combine the Transition LoRA with two reference images: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --pipeline-class-name LTX2TwoStagePipeline \ --image-path ./inputs/start.png ./inputs/end.png \ --lora-path valiantcat/LTX-2.3-Transition-LORA \ --lora-weight-name ltx2.3-transition.safetensors \ --prompt "A fast cinematic transition from the first image to the second image, whip-pan motion, atmospheric lighting, zhuanchang" \ --save-output ``` Some community LoRAs only include weights for transformer blocks. In that case, SGLang logs a concise coverage summary and leaves unmatched LoRA-capable layers on the base model weights. This is expected when the adapter format intentionally omits those layers. ## 5. Practical Tips * Use `--pipeline-class-name LTX2TwoStagePipeline` as the default LTX two-stage quality path. * Use `--pipeline-class-name LTX2TwoStageHQPipeline` when you want the HQ path and have enough VRAM for larger outputs. * Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage. * Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior. * Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions. # LTX2.5 Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2.5 Run LTX-2.5 video + audio generation with SGLang Diffusion. ## 1. Model Introduction [LTX-2.5](https://huggingface.co/Lightricks/LTX-2.5) is an open world model from Lightricks, built for local execution and fine-tuning. Its established use is generating synchronized, high-fidelity video and audio from text, image and video inputs. It is a 22B DiT paired with a Gemma-4-12B text encoder, separate video and audio VAEs, and a vocoder that outputs 48 kHz stereo. Video and audio are denoised jointly in one pass rather than dubbed afterwards, so they stay in sync. Use **`Lightricks/LTX-2.5-Diffusers`** as `--model-path`. **License notice:** LTX-2.5 is released under the LTX-2.x Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://github.com/Lightricks/LTX-2/blob/main/LICENSE.md) before production or commercial use; SGLang support does not grant additional model usage rights. ### 1.1 New in LTX-2.5 Two capabilities have no equivalent in LTX-2 / LTX-2.3: A duration head predicts how long the shot the caption implies should run, and picks the frame count for you. Pass `--auto-duration` instead of `--num-frames`. A diffusion model replaces the convolutional VAE decoder for the latent-to-pixel step. Enable with `--use-diffusion-decoder`. Both are optional and off by default. ### 1.2 Components | Path | Component | Used by | | ------------------------------------------------------------------------------------ | ------------------------------------- | -------------------------------- | | `transformer/` | Distilled DiT (the default) | always | | `transformer_full/` | Full / SFT DiT | `--model-variant dev` | | `vae/` | Convolutional video VAE | encode always; decode by default | | `diffusion_decoder/` | Diffusion video decoder, decoder-only | `--use-diffusion-decoder` | | `latent_upsampler/` | Spatial x2 latent upsampler | `LTX2TwoStagePipeline` | | `duration_head/` | Predicts clip length from the caption | `--auto-duration` | | `audio_vae/`, `vocoder/`, `connectors/`, `text_encoder/`, `tokenizer/`, `scheduler/` | Shared | always | Encoding always uses `vae/`, and both decoders consume the same latents, so the decoder choice does not change anything upstream of it. ## 2. SGLang-diffusion Installation ```bash theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` For platform-specific setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). NATTEN is an optional extra, worth installing only if you plan to use the [diffusion decoder](#4-6-diffusion-decoder) — see that section for why. ## 3. Model Deployment ### 3.1 Basic Configuration ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline ``` On a single high-VRAM GPU no extra flags are needed. **Interactive Command Generator**: pick a target and the features you want; the command updates below. Server-side choices (pipeline class, weights variant, parallelism) go on `sglang serve`, while per-request choices (auto-duration, diffusion decoder, resolution) are listed separately, since they belong on the `sglang generate` call or the request body. ### 3.2 Configuration Tips Choose the pipeline class based on the quality and latency target: | Use case | Pipeline class | Notes | | -------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------- | | One-stage generation | `LTX2Pipeline` | Fastest path. Supports T2V and TI2V, auto-duration and the diffusion decoder. | | Two-stage generation | `LTX2TwoStagePipeline` | Half-resolution base stage, x2 latent upsample, then a short refinement. Pass the **final** resolution. | There is no HQ pipeline class for LTX-2.5, and no `--distilled-lora-path` for either weights variant: LTX-2.5 distils the weights themselves rather than merging a LoRA per stage, so `--ltx2-two-stage-device-mode` (which governs that swap) does not apply either. Every feature on this page — text-to-video, image conditioning, auto-duration, the diffusion decoder, and either weights variant — works with both pipeline classes. Selecting weights: * `--model-variant dev` serves the full / SFT DiT from `transformer_full/`; the default is the distilled one. See [section 4.5](#4-5-the-dev-transformer). ### 3.3 Multi-GPU presets | Target | Recommended server flags | Notes | | ---------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | | 1 high-VRAM GPU | *(no extra flags)* | 960×544 fits comfortably on an H200. | | 1 tight-VRAM GPU | `--quantization fp8` | Halves the DiT and cuts peak memory \~18 GB at unchanged speed. See [section 3.4](#3-4-fp8-quantization). | | 1 very tight GPU | `--dit-layerwise-offload` | Cuts peak memory by roughly 10 GB, at about 4x the wall clock. | | 2 GPUs, long sequences | `--num-gpus 2 --ulysses-degree 2` | Sequence parallel; the memory/long-sequence tool. | | 2 GPUs, large DiT | `--num-gpus 2 --tp-size 2` | Tensor parallel across attention heads. | | 2 GPUs, dev weights | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided and unguided branches across GPUs. Measured 1.77x on denoising (15.1s to 8.5s, 960×544 / 57 frames / 30 steps). | **CFG parallelism does not apply on the default (distilled) path.** That DiT runs unguided, so there is no negative branch to split across GPUs and `--enable-cfg-parallel` buys nothing — the CFG-parallel presets on the LTX-2 / LTX-2.3 page do not carry over. It *is* worth using with `--model-variant dev`, which runs with guidance. ### 3.4 fp8 quantization `--quantization fp8` quantizes the DiT's linear layers as it loads them, so it needs no pre-quantized checkpoint: ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --quantization fp8 ``` At 960×544 / 49 frames the transformer loads in 18.11 GB against 35.37 GB for bf16, and the run peaks at 53.5 GB against 71.1 GB. Denoising time is unchanged: the distilled 8-step path at this size is bound by memory traffic rather than matmul throughput, so fp8 buys headroom rather than speed. Expect a different sample for a given seed. Quantization nudges the denoising trajectory and diffusion amplifies that, so the result differs from bf16 without being worse. ## 4. Model Invocation ### 4.1 Text-to-video with audio ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn, the camera tracking alongside, snow crunching underfoot." \ --save-output ``` Defaults: 960×544, 121 frames, 24 fps. Video and audio are generated jointly and muxed into one MP4. The default DiT is distilled and runs off a fixed 8-sigma schedule rather than a step count, so `--num-inference-steps` and `--guidance-scale` have no effect here. Use [`--model-variant dev`](#4-5-the-dev-transformer) when you want control over either. ### 4.2 Image-to-video ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --image-path ./inputs/start.png \ --prompt "The camera pushes forward as the subject turns toward the light." \ --save-output ``` The conditioning image is re-compressed to match the compression the model was trained against — CRF 18 for LTX-2.5, where LTX-2 / 2.3 use 33. SGLang picks the right one from the checkpoint, so nothing needs to be passed. ### 4.3 Auto-duration NEW LTX-2.5 ships a duration head — a small module that reads the encoded caption and regresses the natural length of the shot it describes. Use it when the prompt implies a duration ("a quick glance" vs "a slow pan across the valley") and you would rather not guess a frame count: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A red fox walking through a snowy forest at dawn." \ --auto-duration \ --save-output ``` The prediction is clamped to `--auto-duration-min-seconds` / `--auto-duration-max-seconds` (default 1–20 s) and snapped to the VAE's temporal grid, so the result is always a valid frame count. It overrides `--num-frames`. ### 4.4 Two-stage (higher quality) Stage 1 runs at half the requested resolution, the latents are upsampled 2x, and a short sigma tail refines at full resolution. Pass the **final** size: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2TwoStagePipeline \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \ --height 1088 --width 1920 \ --save-output ``` Resolution must be divisible by 64. Unlike LTX-2.3, no `--distilled-lora-path` is needed: the LTX-2.5 transformer is already distilled. ### 4.5 The dev transformer LTX-2.5 ships two DiTs. `model_index.json` points at the distilled one; the full / SFT weights live in `transformer_full/` and are deliberately left out of the index. Select them with `--model-variant dev`: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --model-variant dev \ --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \ --num-inference-steps 30 --guidance-scale 3.0 \ --save-output ``` The dev variant is not distilled, so SGLang automatically drops the pinned distilled sigma schedule and re-enables the dynamic shifting that `scheduler/` turns off for the distilled DiT. Unlike the distilled path it *is* driven by a step count and *does* want CFG, so pass `--num-inference-steps` and `--guidance-scale` yourself. Note that `from_pretrained` only fetches what `model_index.json` lists, so a partial snapshot download will not include `transformer_full/` (another 38 GB). ### 4.6 Diffusion decoder NEW LTX-2.5 adds a diffusion-based video decoder as an alternative to the convolutional VAE decoder. Rather than deconvolving the latent it denoises pixels conditioned on a context volume built from it, which recovers detail a convolutional decoder tends to smooth away: ```bash theme={null} sglang generate \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --prompt "A red fox walking through a snowy forest at dawn." \ --use-diffusion-decoder \ --save-output ``` It is a diffusion model in its own right and decodes more slowly than the VAE decoder, so it is off by default — matching upstream, where `LTX2Pipeline` also decodes with the VAE. The offline `generate` command loads the optional decoder automatically when `--use-diffusion-decoder` is present. For an online server, opt into loading the decoder at startup, then select it per request with `use_diffusion_decoder: true`: ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.5-Diffusers \ --pipeline-class-name LTX2Pipeline \ --load-diffusion-decoder ``` This keeps the default server footprint unchanged while still allowing VAE and diffusion-decoder requests to share one server. When GPU memory is constrained, `--cpu-offload-components diffusion_decoder` keeps the optional decoder on CPU between uses. **Install NATTEN for this decoder.** Its stages run 3D neighborhood attention, and SGLang uses NATTEN's fused `na3d` kernel for it when the package is present. NATTEN is *not* a dependency of `sglang[diffusion]`: without it the decoder falls back to a compiled FlexAttention block mask. The two agree to bf16 rounding, but the fallback is roughly **5x slower** on the decoder's largest attention grid, and has to build the mask on top of that. NATTEN ships prebuilt wheels pinned to a specific torch and CUDA build, so install the one matching your environment rather than a bare version — check your combination at [natten.org](https://natten.org). For torch 2.11 / CUDA 13.0, for example: ```bash theme={null} uv pip install natten==0.21.6+torch2110cu130 -f https://whl.natten.org/ ``` Nothing else changes if you skip it: the decoder still produces the same video, just slower. # LingBot Video MoE Source: https://docs.sglang.io/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE Serve the native LingBot Video MoE 30B-A3B text-to-video model with SGLang Diffusion. ## 1. Model Introduction [LingBot Video MoE 30B-A3B](https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b) is a text-to-video mixture-of-experts model served through a native SGLang Diffusion pipeline. Its main advantage is the MoE economics: 30B total parameters with 3B active per token, so a plain `sglang serve` invocation covers the released 480×480, 81-frame profile without parallelism flags. Prompting is the distinctive part of this checkpoint: it expects a structured JSON caption rather than an unexpanded natural-language prompt, passed as the request's `prompt` string. A compact 17-frame, 12-step smoke-test profile is available for quick validation next to the released 40-step profile. The public checkpoint served by the pipeline: | Model ID | Task | Default output | | ------------------------------------ | ------------- | ---------------------------- | | `robbyant/lingbot-video-moe-30b-a3b` | Text to video | 480x480, 81 frames at 16 FPS | The checkpoint expects a structured JSON caption rather than an unexpanded natural-language prompt. The JSON is passed as the request's `prompt` string; it is not an `extra_params` object. ## 2. Installation Install SGLang with the diffusion dependencies: ```bash Command theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` See the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for platform-specific setup. ## 3. Serve LingBot Video MoE Start the server with the Hugging Face model ID: ```bash Command theme={null} sglang serve \ --model-path robbyant/lingbot-video-moe-30b-a3b \ --port 30010 ``` ## 4. Generate a video The following request uses the compact 17-frame, 12-step smoke-test profile. Use the model defaults of 81 frames and 40 steps for the released generation profile. ```python Python theme={null} import json import time from pathlib import Path import requests base_url = "http://127.0.0.1:30010" prompt = json.dumps( { "comprehensive_description": { "scene_content_description": ( "A small silver robot arm on a white table slowly reaches " "toward a red cube. The background is a softly lit laboratory wall." ), "camera_movement_description": ( "The camera is static at eye level in a medium shot." ), }, "camera_info": { "color": "Neutral", "frame_size": "Medium", "shot_type_angle": "Eye level", "lens_size": "Medium", "composition": "Center", "lighting": "Soft light", "lighting_type": "Artificial light", }, "world_knowledge": [], "prominent_elements": [ { "name": "robot arm", "description": "A small silver robot arm with a two-finger gripper.", "actions": [ { "timestamp": "[0.0s - 1.0s]", "action": "reaches toward the red cube", } ], "location": "center of the frame", "relative_size": "dominant", "shape_and_color": "articulated silver metal arm", "texture": "brushed metal", "appearance_details": "two-finger gripper and visible joints", "relationship": "reaching toward the red cube on the table", "orientation": "upright, base on the table", "pose": "reaching", } ], }, separators=(",", ":"), ) response = requests.post( f"{base_url}/v1/videos", json={ "model": "robbyant/lingbot-video-moe-30b-a3b", "prompt": prompt, "size": "640x384", "num_frames": 17, "fps": 16, "num_inference_steps": 12, "guidance_scale": 6.0, "flow_shift": 3.0, "seed": 0, }, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) video = requests.get( f"{base_url}/v1/videos/{video_id}/content", timeout=300, ) video.raise_for_status() Path("lingbot_video_moe.mp4").write_bytes(video.content) ``` ## 5. Request constraints * `num_frames` must be `1` or `4n+1`; examples include 17 and 81. * Width and height must both be multiples of 16. * The native defaults are `guidance_scale=6.0`, `flow_shift=3.0`, `num_inference_steps=40`, and `fps=16`. * Keep the prompt as serialized JSON. Raw free text is outside the checkpoint's expected caption format. # LingBot World Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World ## 1. Model Introduction [LingBot World](https://huggingface.co/robbyant/lingbot-world-fast-diffusers) is a realtime camera-controlled video world model. It keeps a live causal session, applies prompt and camera events between chunks, reuses DiT/VAE state, and streams decoded frames instead of finishing a bounded clip before returning. Choose it for interactive exploration and continuous control, not one-shot cinematic generation. The causal window makes long sessions practical but gives up the global bidirectional context available to offline Wan or LTX pipelines; session state and cache policy are therefore part of the serving contract. ## 2. Deployment ```bash Command theme={null} sglang serve \ --model-path robbyant/lingbot-world-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 4 \ --ulysses-degree 4 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false ``` ### Optional KV-Cache Compression Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing `quant-videogen` as described in the quantization guide and adding `--kv-cache-quant int4` to the server command. The current and most recent completed chunks remain in BF16. See [Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization) for the algorithm, tuning options, measured memory-latency tradeoff, and support limits. ## 3. Realtime WebUI The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior. ```bash Command theme={null} python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui ``` Open `http://127.0.0.1:18080` and use: ```text Example theme={null} ws://127.0.0.1:30000/v1/realtime_video/generate ``` ## 4. HTTP and WebSocket API LingBot World uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running. ### Endpoints | API | Method | Purpose | Notes | | ----------------------------- | ----------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. | | `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. | ### `init` message Send this MessagePack map immediately after the WebSocket opens. | Parameter | Type | Required | Meaning | | ------------------------------------- | --------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | Must be `"init"`. | | `model` | string | No | Model id. Leave empty to use the served model. | | `prompt` | string | Yes | Text prompt for the initial scene and motion style. | | `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. | | `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. | | `fps` | number | Yes | Target playback FPS for the generated stream. | | `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. | | `seed` | integer | No | Random seed for deterministic sampling. | | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | | `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | | `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` sends raw RGB; omit for lossless delta-gzip RGB. | | `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. | | `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. | | `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. | | `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. | | `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. | | `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. | | `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. | | `frame_interpolation_model_path` | string | No | Optional RIFE model path. | | `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. | ### Runtime `event` messages After `init`, send MessagePack event maps to update the live session. | Parameter | Type | Required | Meaning | | ---------- | -------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | Must be `"event"`. | | `kind` | `"prompt"` or `"camera_actions"` | Yes | Runtime event kind. | | `payload` | string or object/list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. | | `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. | `camera_actions` supports two payload modes: | Mode | Payload shape | Meaning | | ------ | ------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. | | State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. | Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls. ### Server messages | Message | Payload | Meaning | | ------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. | | `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. | | `error` | MessagePack map | Server-side validation or generation error. | ### Minimal client sketch ```python Python theme={null} import msgspec.msgpack import websocket ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate") ws.send_binary(msgspec.msgpack.encode({ "type": "init", "prompt": "A quiet rainy London alley, stable camera motion.", "first_frame": open("reference.jpg", "rb").read(), "size": "832x480", "fps": 25, "num_frames": 9, "num_inference_steps": 4, "guidance_scale": 1, "realtime_output_format": "webp", "output_compression": 95, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]}, })) ``` ## 5. Consistency LingBot World uses raw-frame websocket GT plus per-chunk latency guards for consistency checks. ## 6. Notes * Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. * Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. * Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. # LingBot World 2.0 Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World-2.0 ## 1. Model Introduction [LingBot World 2.0](https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast-diffusers) is the 14B realtime camera-controlled world model in the LingBot family. It generates continuously from a live causal session, accepts control updates between chunks, and reuses DiT and VAE state for incremental frame delivery. Choose 2.0 when the larger model and interactive KV-window path justify an 8-GPU deployment. Like the earlier LingBot release, it prioritizes responsive control and bounded causal state over the full-clip global context of offline video diffusion. ## 2. Deployment ```bash Command theme={null} export SGLANG_LINGBOT_LAZY_VAE_ENCODE_BLACK_FRAMES=60 export SGLANG_LINGBOT_ENABLE_INTERACTIVE_KV_WINDOW=true sglang serve \ --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 8 \ --ulysses-degree 8 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false \ --vae-config.use-parallel-decode true \ --vae-config.parallel-decode-mode spatial \ --enable-torch-compile false ``` ### Optional KV-Cache Compression Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing `quant-videogen` as described in the quantization guide and adding `--kv-cache-quant int4` to the server command. The current and most recent completed chunks remain in BF16. See [Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization) for the algorithm, tuning options, measured memory-latency tradeoff, and support limits. ## 3. Realtime WebUI The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior. ```bash Command theme={null} python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui ``` Open `http://127.0.0.1:18080` and use: ```text Example theme={null} ws://127.0.0.1:30000/v1/realtime_video/generate ``` ## 4. HTTP and WebSocket API LingBot World 2.0 uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running. ### Endpoints | API | Method | Purpose | Notes | | ----------------------------- | ----------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. | | `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. | ### `init` message Send this MessagePack map immediately after the WebSocket opens. | Parameter | Type | Required | Meaning | | ------------------------------------- | --------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `type` | string | Yes | Must be `"init"`. | | `model` | string | No | Model id. Leave empty to use the served model. | | `prompt` | string | Yes | Text prompt for the initial scene and motion style. | | `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. | | `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. | | `fps` | number | Yes | Target playback FPS for the generated stream. | | `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. | | `seed` | integer | No | Random seed for deterministic sampling. | | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | | `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | | `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` sends raw RGB; omit for lossless delta-gzip RGB. | | `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. | | `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. | | `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. | | `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. | | `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. | | `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. | | `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. | | `frame_interpolation_model_path` | string | No | Optional RIFE model path. | | `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. | ### Runtime `event` messages After `init`, send MessagePack event maps to update the live session. | Parameter | Type | Required | Meaning | | ---------- | ------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | string | Yes | Must be `"event"`. | | `kind` | `"prompt"`, `"camera_actions"`, or `"composite_input"` | Yes | Runtime event kind. | | `payload` | string, object, or list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. For `composite_input`, a map containing `input_types` plus each named input. | | `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. | `camera_actions` supports two payload modes: | Mode | Payload shape | Meaning | | ------ | ------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. | | State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. | Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls. Use `composite_input` when multiple runtime inputs should be sampled together, such as updating the prompt and camera controls in one event. ### Server messages | Message | Payload | Meaning | | ------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. | | `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. | | `error` | MessagePack map | Server-side validation or generation error. | ### Minimal client sketch ```python Python theme={null} import msgspec.msgpack import websocket ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate") ws.send_binary(msgspec.msgpack.encode({ "type": "init", "prompt": "A quiet rainy London alley, stable camera motion.", "first_frame": open("reference.jpg", "rb").read(), "size": "832x480", "fps": 25, "num_frames": 9, "num_inference_steps": 4, "guidance_scale": 1, "realtime_output_format": "webp", "output_compression": 95, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]}, })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "prompt", "event_id": 2, "payload": "A quiet snowy Tokyo alley, stable camera motion.", })) ws.send_binary(msgspec.msgpack.encode({ "type": "event", "kind": "composite_input", "event_id": 3, "payload": { "input_types": ["prompt", "camera_actions"], "prompt": "A quiet neon Shanghai alley, stable forward camera motion.", "camera_actions": [["w"], ["w"], []], }, })) ``` ## 5. Consistency LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for consistency checks. ## 6. Notes * Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. * Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. * Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. # LongLive 2.0 Source: https://docs.sglang.io/cookbook/diffusion/LongLive/LongLive-2.0 Serve LongLive 2.0 distilled text-to-video and image-to-video models with SGLang-diffusion. ## 1. Model Introduction [LongLive 2.0](https://nvlabs.github.io/LongLive/LongLive2/) is NVIDIA's 4-step text/image-to-video model distilled from Wan2.2-TI2V-5B. Its main strength is extending few-step causal generation across prompt changes, so a single request can produce multi-shot sequences without paying a full diffusion schedule for every shot. Choose it for low-step long or multi-shot generation rather than maximum one-shot fidelity. The SGLang path uses a Diffusers conversion of the official weights, and scene continuity still depends on prompt-block and sink settings; validate transitions on the target storyboard. The model weights use the NVIDIA Open Model License. See the [paper](https://arxiv.org/abs/2605.18739) and [GitHub repository](https://github.com/NVlabs/LongLive) for training details. ## 2. SGLang-diffusion Installation Please refer to the [official SGLang-diffusion installation guide](/docs/sglang-diffusion/installation) for installation instructions. ## 3. Deployment ```bash Command theme={null} sglang serve --model-path Rabinovich/LongLive-2.0-5B-Diffusers ``` In `auto` mode, GPUs with at least 60 GiB available keep the DiT, text encoder, and VAE resident. This uses about 44 GiB on H200 for the 832x480 preset and avoids moving encoder and decoder layers from host memory on every request. Smaller GPUs retain the layerwise-offload defaults. If the GPU runs out of memory, move the text encoder, VAE, and DiT to CPU between stages: ```bash Command theme={null} sglang serve \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --dit-cpu-offload \ --text-encoder-cpu-offload \ --vae-cpu-offload ``` `Rabinovich/LongLive-2.0-5B-Diffusers` is the Diffusers-format conversion of the official `Efficient-Large-Model/LongLive-2.0-5B` weights. ## 4. Generation ### 4.1 Single prompt Generate one clip without starting a server: ```bash Command theme={null} sglang generate \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --prompt "A quiet street at dusk" \ --num-frames 61 \ --save-output \ --output-path outputs ``` 61 frames is 16 latent frames, which is two causal blocks of 8. ### 4.2 Multi-shot long video Multi-shot prompts are sampling parameters, so pass them through the Python API: ```python Python theme={null} from sglang import DiffGenerator gen = DiffGenerator.from_pretrained("Rabinovich/LongLive-2.0-5B-Diffusers") result = gen.generate(sampling_params_kwargs={ "shot_prompts": [ "A husky walks down a sunlit hallway.", "The husky turns and looks at the camera.", "Two dogs play together on a carpet.", ], "chunks_per_shot": 4, "num_frames": 381, # 3 shots x 4 chunks x 8 = 96 latent frames -> 381 frames "scene_cut_prefix": "The scene transitions. ", "multi_shot_sink": True, "multi_shot_rope_offset": 8.0, "save_output": True, "output_path": "outputs", }) ``` Each shot runs for `chunks_per_shot` causal blocks before the next prompt is used. The multi-shot defaults mirror the original LongLive prompt-block settings. ### 4.3 Key parameters These are SGLang request parameters. Original LongLive configs use latent-frame `num_output_frames`; SGLang exposes output-video `num_frames`. * `num_frames`: 61 in the examples. This maps to 16 latent frames, while the original release config defaults to 128 latent frames. * `num_inference_steps`: 4, matching original `sampling_steps`. * `guidance_scale`: 1.0, matching the original inference config. * `height` / `width`: 704 / 1280 by default, matching original latent H/W 44 / 80 with 16x spatial compression. * `shot_prompts`, `chunks_per_shot`, `scene_cut_prefix`, `multi_shot_sink`, and `multi_shot_rope_offset`: SGLang request fields for the original prompt-block and multi-shot behavior. ### 4.4 Image-to-video Pass a first frame with `--image-path` to condition the clip on an image: ```bash Command theme={null} sglang generate \ --model-path Rabinovich/LongLive-2.0-5B-Diffusers \ --prompt "A quiet street at dusk" \ --image-path first_frame.png \ --num-frames 61 \ --save-output \ --output-path outputs ``` The image is used as the first-frame condition. ## 5. Notes * `num_frames` must map to a whole number of causal blocks. The latent frame count is `(num_frames - 1) / 4 + 1` and must be divisible by 8. For example, 61, 125, and 189 frames give 16, 32, and 48 latent frames. * SGLang supports T2V sizes 1280x704, 704x1280, 832x480, and 480x832. * I2V request images follow the Wan TI2V preprocessing path in SGLang. This is different from the original LongLive dataset resize path. * For multi-shot runs, set `num_frames` to match `len(shot_prompts) * chunks_per_shot * 8` latent frames, that is `num_frames = (len(shot_prompts) * chunks_per_shot * 8 - 1) * 4 + 1`. # MOVA Source: https://docs.sglang.io/cookbook/diffusion/MOVA/MOVA ## 1. Model Introduction [MOVA](https://github.com/OpenMOSS/MOVA) generates video and audio together with an asymmetric dual-tower model connected by bidirectional cross-attention. Its strongest use cases are speaking subjects, visible sound-producing events, and scenes where ambient audio must track the picture rather than be synthesized by a later cascade. The public 360p and 720p checkpoints both generate up to 8 seconds. Choose 360p for the lighter deployment and 720p for output resolution; MOVA is less suitable when the task needs long-form continuity or the richer image/video/audio reference conditioning provided by H3. | Checkpoint | Best fit | Output limit | | ------------------------- | ------------------------------------------------- | ----------------------- | | `OpenMOSS-Team/MOVA-360p` | Faster, lower-memory joint audiovisual generation | Up to 8 seconds at 360p | | `OpenMOSS-Team/MOVA-720p` | Higher-resolution lip-sync and environment audio | Up to 8 seconds at 720p | ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration MOVA supports both online serving and CLI generation modes. The recommended launch configurations vary by hardware and resolution. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--num-gpus`: Number of GPUs to use * `--tp`: Tensor parallelism size (should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--ring-degree`: The degree of ring attention-style SP in USP * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--adjust-frames`: Whether to adjust frames automatically (set to `false` for MOVA) * `--enable-torch-compile`: Enable torch.compile for faster inference ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 CLI Generation (sglang generate) ```bash Command theme={null} sglang generate \ --model-path OpenMOSS-Team/MOVA-720p \ --prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, \ framed by wooden furniture and a filled bookshelf. \ Quiet room acoustics underscore his measured tone as he delivers his remarks. \ At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \ --image-path "" \ --adjust-frames false \ --num-gpus 8 \ --ring-degree 2 \ --ulysses-degree 4 \ --num-frames 193 \ --fps 24 \ --seed 67 \ --num-inference-steps 25 \ --enable-torch-compile \ --save-output ``` ### 4.2 Generate a Video ```bash Command theme={null} curl -X POST "http://0.0.0.0:30002/v1/videos" \ -F "prompt=A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \ -F "input_reference=@" \ -F "size=640x352" \ -F "num_frames=193" \ -F "fps=24" \ -F "seed=67" \ -F "guidance_scale=5.0" \ -F "num_inference_steps=25" \ -o create_video.json ``` ### 4.3 Advanced Usage #### 4.3.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path OpenMOSS-Team/MOVA-720p ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path OpenMOSS-Team/MOVA-720p ``` #### 4.3.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark ### 5.1 Speedup Benchmark #### 5.1.1 Generate a video Test Environment: * Hardware: NVIDIA H200 x 8 * git revision: 443b1a8 * Model: OpenMOSS-Team/MOVA-720p **Server Command**: ```bash Command theme={null} sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \ --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \ --tp 1 --enable-torch-compile ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --task image-to-video --dataset vbench --num-prompts 1 --max-concurrency 1 \ --port 30002 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: image-to-video Model: OpenMOSS-Team/MOVA-720p Dataset: vbench -------------------------------------------------- Benchmark duration (s): 590.76 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 590.7549 Latency Median (s): 590.7549 Latency P99 (s): 590.7549 -------------------------------------------------- Peak Memory Max (MB): 74996.00 Peak Memory Mean (MB): 74996.00 Peak Memory Median (MB): 74996.00 ============================================================ ``` #### 5.1.2 Generate videos with high concurrency **Server Command**: ```bash Command theme={null} sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \ --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \ --tp 1 --enable-torch-compile ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \ --port 30002 ``` # MiniMax-H3 Source: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3 Run native MiniMax-H3 video-and-audio generation with SGLang Diffusion. ## 1. Quick start Install with `uv pip install "sglang[diffusion]" --prerelease=allow`, then choose a verified recipe below. Setup changes the deployment; Server and Request expose orthogonal startup and sampling choices. The generated Server command already includes the recommended encoder policy. Change a Server option only for a deliberate trade-off; Request options do not reload the model. The Docker form installs the platform-specific diffusion extra from the source bundled in the image. For conditioned requests, set **Host media directory** under **Variables**; the builder mounts it read-only at `/data/minimax-h3`. AMD currently offers the Python form, while NVIDIA also offers Docker. To use ModelScope through the same normal `sglang serve` path, prefix the copied command with `SGLANG_USE_MODELSCOPE=true` and replace the model path with `MiniMax/MiniMax-H3`. Keep the selected variant and topology flags unchanged. For platform-specific installation details, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 2. Model capabilities [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) is a native joint video-and-audio model for text-to-video-and-audio, first/last-frame control, and multimodal reference conditioning. Its main strength is producing the picture and stereo soundtrack together, so speech, music, ambient sound, and visible events can stay aligned without a separate audio-generation pass. Choose H3 when synchronized audiovisual output or reference-driven generation matters more than a lightweight deployment. The released recipe targets a 768-pixel short edge at 24 fps for 4–15 seconds, and its capabilities are split across two checkpoint partitions; serving every mode therefore requires separate FL2VA and Ref2VA deployments. | Task | `task` value | Conditioning | | ----------------------------------- | ------------ | ---------------------------------- | | Text to video and audio | `t2va` | Text prompt only | | First/last frame to video and audio | `fl2va` | First frame, last frame, or both | | Reference to video and audio | `ref2va` | Image, video, and audio references | Video-to-video (V2V) is a supported `ref2va` use case, not a fourth task value. Run the `Ref2VA` partition and provide a video reference in `conditions`. Use the selected Hub's root model ID: `MiniMaxAI/MiniMax-H3` on Hugging Face or `MiniMax/MiniMax-H3` on ModelScope. Select the checkpoint variant with `--model-variant`: `fl2va` serves both `t2va` and `fl2va`, while `ref2va` serves reference-conditioned requests. SGLang owns the checkpoint-directory mapping; do not point `--model-path` at a manually downloaded subdirectory. Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights. ## 3. Deployment details The builder accepts legal custom GPU counts and topologies, marking them **Unverified** until the exact recipe has completed end-to-end validation. Static H3 head or partition violations disable Copy before they reach `sglang serve`. For a four-card H200 host, keep the full BF16/FP32 model resident by default. The model fits without FSDP, so this path avoids the per-block parameter all-gathers of the memory-oriented FSDP profile: ```bash 4×H200 resident theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Pure Ulysses4 is also the faster measured topology on H200, not just a capacity default. The 4×H100 TP2 + Ulysses2 recipe below fits on 141 GB H200 cards, but it replaces the Ulysses all-to-all exchange with two per-block tensor-parallel all-reduces and measured slower end-to-end, at about 30 GB lower peak memory per GPU. See the **H200 topology comparison** in the Benchmarks section for the measured numbers; treat TP2 + Ulysses2 on H200 as a deliberate memory trade, not a latency default. For 4×H100 80 GB, balance the large packed activation with resident weight sharding. TP2 + Ulysses2 was the fastest measured lossless topology while the Qwen encoder still folds across all four GPUs: ```bash 4×H100 fastest theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Pure Ulysses4 could not keep the full pipeline resident on 80 GB H100s. Use `--tp-size 4 --ulysses-degree 1` when lower resident memory matters more than the last few percent of latency. FSDP remains a verified capacity option, but its per-block weight all-gathers do not make it the H100 speed default: ```bash 4×H100 FSDP capacity theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference true \ --port 30010 ``` For a two-card RTX 5090 host, use TP2 and keep 20 DiT blocks resident. Layerwise placement is lossless: it changes parameter placement and transfer scheduling, not the BF16/FP32 denoising or VAE math. This is the fastest measured 32 GB operating point: ```bash 2×RTX 5090 fastest lossless theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 2 \ --tp-size 2 \ --ulysses-degree 1 \ --encoder-parallel auto \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 20 \ --enable-torch-compile false \ --port 30010 ``` The DiT residency and prefetch knobs apply only to the repeatedly executed DiT blocks. The text encoder and the video VAE decoder blocks use one-layer prefetch with zero resident layers. The video VAE encoder stays resident because its indexed down blocks cannot host executable layerwise hooks; the roughly 577 MiB audio VAE also stays resident because offloading it only adds transfer overhead. This exact recipe was validated on 2× RTX 5090 (32 GB each) and a 377 GiB host; use a 384 GiB-class machine. The latency and memory comparison is collected in the benchmark section below. For a single 24 GB consumer card (RTX 4090), stream the DiT and text encoder and quantize DiT linear layers online with `kitchen_int8`. Keep `vae` out of `--layerwise-offload-components`: putting the VAE decoder in layerwise offload re-streams about 9 GiB on each of 167 decode tiles. Default attention stays `fa` (exact). Approximate backends are opt-in; see [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid). Install `comfy-kitchen` first (`pip install comfy-kitchen`). ```bash 1×RTX 4090 24GB theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --quantization kitchen_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` The same flags work on `sglang serve`. Drop `--quantization` for the BF16 baseline; everything else stays identical. GPU peak stays about 18 GB either way because streaming offload is set by the offload buffers and VAE decode, not the weight dtype. ### Pre-quantized GGUF transformer Use `--transformer-weights-path` to replace only the DiT with a GGUF file; the base repository continues to provide the text encoder, VAEs, scheduler, and tokenizers. Do not also pass `--quantization gguf`. ```bash 1×RTX 5090 Q4_K_M theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --port 30010 ``` The loader also recognizes pruned checkpoints that replace the timestep MLP with `adaln_t_table`. Repositories containing both FL2VA and Ref2VA variants need a full file reference: ```bash Pruned FL2VA Q4_K theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --enable-torch-compile false \ --port 30010 ``` The linear adapter reuses SRT's GGUF type definitions and CUDA dequantization, then runs the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM regime and are slower at diffusion sequence lengths. TP is supported when each row-parallel input shard remains GGML-block aligned; incompatible degrees fail during model construction. FSDP, LoRA merging, and the separate MiniMax-H3 AdaLN cache flags are not compatible with packed GGUF weights. The first launch downloads the model through the selected Hub. If the Hugging Face repository requires authentication, export a Hugging Face token in the server environment. For MiniMax-H3, `--performance-mode speed` deliberately keeps the DiT eager. The current `torch.compile` path changes the model's numerical output, so it is not enabled implicitly by any recommended lossless preset. An explicit `--enable-torch-compile true` remains available for controlled experiments, but it should not be used to generate consistency ground truth. ### AdaLN-pruned safetensors transformers [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models) publishes smaller DiT-only checkpoints that replace the original AdaLN branches with an interpolated curve table. Select one file explicitly; the base model still supplies the text encoder and VAEs. ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ Comfy-Org/MiniMax-H3/diffusion_models/minimax_h3_fl2va_pruned_bf16.safetensors \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --performance-mode speed \ --port 30010 ``` The pruned checkpoint is approximate and is therefore rejected by `quality="high"`, which remains limited to the audited official BF16 DiT. The Comfy `pruned_fp8_scaled` FL2VA and Ref2VA files are also supported. Their per-layer markers are detected automatically; do not add `--quantization`: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ Comfy-Org/MiniMax-H3/diffusion_models/minimax_h3_fl2va_pruned_fp8_scaled.safetensors \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --performance-mode speed \ --port 30010 ``` SGLang uses its native static-activation FP8 linear path when the checkpoint stores input scales, and automatically uses dynamic activation scaling for Comfy FP8 exports that omit them. The checkpoint above marks `fc2` for full-precision matrix multiplication, so SGLang retains its FP8 storage but materializes and scales one compute-dtype `fc2` matrix for each call. This preserves the checkpoint's mixed execution contract and low resident weight memory, but that part is slower than a fully quantized FP8 GEMM. TP, Ulysses/Ring sequence parallelism, and component/layerwise offload are supported; FSDP inference is rejected. The `pruned_int8_convrot` files use the same override path and are detected automatically. Install `comfy-kitchen`, replace the FP8 filename above with `minimax_h3_fl2va_pruned_int8_convrot.safetensors`, and still omit `--quantization`. SGLang loads their serialized INT8 weights and row scales into the fused ConvRot kernel without requantizing them. TP1/2/4 and sequence or layerwise offload are supported; TP8 violates the checkpoint's 256-element ConvRot group boundary, and FSDP is rejected. Self-describing community MXFP8 files use the same `--transformer-weights-path ` form and need no quantization flag. SGLang reads their global format metadata, keeps unmarked H3 layers in their original dtype, and reuses SRT's MXFP8 dense kernels. The selected SRT backend must support MXFP8 on the target GPU; FSDP is rejected for this mixed per-layer layout. W4A8 ConvRot DiT files use the same flagless flow. With `comfy-kitchen>=0.2.27`, pass a file such as `starsfriday/MiniMax-H3-w4a8/minimax_h3_fl2va_pruned_w4a8_mixed.safetensors` to `--transformer-weights-path`; SGLang reads the serialized per-layer metadata and packed INT4 tensors automatically. Do not add `--quantization`. TP remains subject to each row-parallel shard preserving the checkpoint's ConvRot group boundary, and FSDP is rejected. W4A4 ConvRot files are also detected from their layer metadata. Install a current `comfy-kitchen`, then pass a full or pruned FL2VA / Ref2VA file such as `Merserk/MiniMax-H3-INT4-ConvRot/minimax_h3_fl2va_pruned_int4_convrot.safetensors` to `--transformer-weights-path`. The packed weights stay INT4 and the runtime honors each layer's activation mode; omit `--quantization`. Mixed exports use the same command: SGLang dispatches each marked layer to its serialized W4A4 or INT8 ConvRot kernel instead of applying one global method. Comfy NVFP4 DiTs reuse SGLang's ModelOpt NVFP4 backend, which requires CUDA compute capability 10.0 or newer. Pass a pruned FL2VA / Ref2VA file such as `Abiray/Minimax-H3-nvfp4-INT4-INT8-Convrot/MiniMax_H3_FL2VA_pruned_nvfp4.safetensors` to `--transformer-weights-path` and omit `--quantization`. SGLang infers the packed group size and Comfy scale layout from the checkpoint; FSDP is rejected. Mixed files may mark selected linears as `int8_tensorwise` or dynamic/static FP8; SGLang dispatches those layers to their serialized Kitchen INT8 or native FP8 path automatically. ### Advanced: precomputed AdaLN cache The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about 13B H3 parameters are AdaLN branches whose outputs can be precomputed for inference. The public base checkpoint contains the original branches, not a ready-to-use cache. SGLang therefore keeps the standard path as the default. This is an experimental deployment path. It is intentionally disabled unless you provide an explicitly generated cache; end-to-end numerical and peak-memory validation remains required before using it in production. When an inference-only deployment has a fixed sampling schedule, build a cache from the already materialized transformer directory on CUDA, then pass it to the usual `sglang serve` command. This does not alter the denoising formula: the cache stores the BF16 outputs of the original AdaLN linears. ```bash Command theme={null} python -m sglang.multimodal_gen.tools.build_minimax_h3_adaln_cache \ --transformer-path "$TRANSFORMER_PATH" \ --model-variant fl2va \ --mode t2va \ --num-inference-steps 50 \ --flow-shift 12 \ --audio-flow-shift 3 \ --output /models/minimax-h3-fl2va-adaln-50step.safetensors sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --minimax-h3-adaln-cache-path /models/minimax-h3-fl2va-adaln-50step.safetensors \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --port 30010 ``` `$TRANSFORMER_PATH` is the `FL2VA/transformer` or `Ref2VA/transformer` directory in the normal SGLang/Hugging Face snapshot; the builder never downloads a second copy. A cache only covers the scheduler settings used to create it, including its mode, step count, flow shifts, and condition noise values. SGLang rejects a request outside that coverage instead of silently changing conditioning. Cache mode supports the matching unquantized checkpoint only. ## 4. Generate video and audio MiniMax-H3 uses the asynchronous OpenAI-compatible video endpoint. Choose a generation mode below, submit a job, poll its status, and then download the completed MP4. MiniMax-H3 supports output durations from 4 through 15 seconds, inclusive. The following request keeps the verified 5-second profile at a 768-pixel short edge. MiniMax-H3 resolves the aligned output canvas and frame count from `target`. ```bash Command theme={null} video_id=$( curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.", "seconds": 5, "task": "t2va", "conditions": [], "target": { "short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 5.0 }, "num_outputs_per_prompt": 1, "num_inference_steps": 50, "flow_shift": 12.0, "audio_flow_shift": 3.0, "seed": 1101 }' | jq -r '.id' ) while true; do status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${video_id}" | jq -r '.status') [ "$status" = "completed" ] && break [ "$status" = "failed" ] && exit 1 sleep 1 done curl -sS -L "http://127.0.0.1:30010/v1/videos/${video_id}/content" \ -o minimax-h3-t2va.mp4 ``` The output contract is an MP4 containing H.264 video at 24 fps and one AAC stereo audio stream at 32 kHz. For `fl2va`, provide one or two image conditions with role `keyframe`. The supported frame-index sets are `[0]`, `[-1]`, and `[0, -1]`. The following request uses one server-local first frame. Use `frame_index: -1` for a last frame, or include both entries for first-and-last conditioning. Choose FL2VA when the supplied image should be the actual first or last frame of the generated clip. Use image-based Ref2VA instead when the image should guide identity, style, or composition without being preserved as an endpoint; Ref2VA may recompose or crop the reference. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "The supplied frame continues with calm, natural motion and synchronized ambient sound.", "seconds": 5, "task": "fl2va", "conditions": [ { "type": "image", "uri": "file:///data/minimax-h3/first-frame.png", "role": "keyframe", "frame_index": 0 } ], "target": { "short_edge": 768, "aspect_ratio": "auto", "duration_seconds": 5.0 }, "num_outputs_per_prompt": 1, "num_inference_steps": 50, "flow_shift": 12.0, "audio_flow_shift": 3.0, "seed": 2101 }' ``` V2V uses the reference-conditioning weights. Launch the server with `--model-variant ref2va`, keep the request `task` set to `ref2va`, and provide a video reference in `conditions`. There is no separate `v2v` task value. Use `type: "video"` when the input may be silent. If the file has a soundtrack, H3 also uses it as an audio reference. Use `type: "video_audio"` only when both streams are required; that form rejects an input without audio. The prompt tag for the visual stream is ` For `ref2va`, first launch the reference-conditioning capability with `--model-variant ref2va`, then provide conditions with role `reference`. Image, video, and audio references can be combined. Material tags in the prompt use the one-based order for each modality. An image condition here is semantic reference material rather than a pixel-aligned first frame. Use the FL2VA tab when animating a screenshot from that exact starting composition. ```bash Command theme={null} curl -sS -X POST http://127.0.0.1:30010/v1/videos \ -H "Content-Type: application/json" \ -d '{ "model": "MiniMaxAI/MiniMax-H3", "prompt": "Use as the visual subject and Poll and download any conditioned request with the same job-status and content endpoints used in the T2VA example. Server-local `file://` URIs must refer to files visible inside the SGLang server environment. ## 5. LoRA recipes H3 accepts both native fused adapters and standard Diffusers/PEFT adapters. Native adapters target modules such as `blocks.*.attn.qkv_proj`; PEFT adapters may instead provide separate `to_q`, `to_k`, and `to_v` projections and the `default` adapter namespace. SGLang normalizes both layouts. The following FL2VA adapters have distinct purposes: | Recipe | Repository and pinned file | Request setting | Prompt requirement | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------- | | Recommended speed/quality balance | [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora), `minimax_h3_turbo_v4_step600_ema.safetensors` | `num_inference_steps: 9` (8 denoiser evaluations), `lora_scale: 1.0` | None | | Most aggressive speed preset (standard PEFT layout) | [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo), `minimax_h3_fl2v_turbo_4step_v0.1.safetensors` | `num_inference_steps: 5` (4 denoiser evaluations), `lora_scale: 1.0`, `lora_alpha: 8` | None | | Realistic people style | [`fal/MiniMax-H3-Realism-People-LoRA`](https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA), `h3-realism-people-t2v-i2v-r2v.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.7` | Include `r34l1sm` in the prompt | The H3 request field controls the number of sigma grid points, including the terminal zero; the denoising loop therefore runs one fewer model evaluation. This is why an adapter described as 8-step uses `9`, and a 4-step adapter uses `5`, in the request. All three use the same launch shape. Pinning the filename is required for repositories that publish multiple revisions, and is also recommended for a reproducible single-file recipe: ```bash Command theme={null} LORA_REPO=larryvrh/MiniMax-H3-Turbo-Lora LORA_FILE=minimax_h3_turbo_v4_step600_ema.safetensors LORA_NAME=h3-turbo-v4 LORA_SCALE=1.0 LORA_ALPHA_ARGS=() # LightX2V only: LORA_ALPHA_ARGS=(--lora-alpha 8) sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --ulysses-degree 4 \ --performance-mode speed \ --lora-path "$LORA_REPO" \ --lora-weight-name "$LORA_FILE" \ --lora-nickname "$LORA_NAME" \ --lora-scale "$LORA_SCALE" \ "${LORA_ALPHA_ARGS[@]}" \ --lora-merge-mode auto \ --port 30010 ``` `auto` merges an adapter into ordinary resident weights to avoid per-step LoRA matmuls, but keeps the dynamic path for FSDP-sharded weights where a full gather can increase peak memory. Use `dynamic` when one resident server must switch repeatedly between base and LoRA output. Use the filename, scale, and request schedule from the table together. The 4-evaluation LightX2V recipe is the more aggressive latency/quality tradeoff. Its checkpoint has rank 128 but omits the training alpha from both the file and repository metadata, so `--lora-alpha 8` is required to reproduce the author's reference implementation. Start with the Larry 8-evaluation recipe when preserving fine visual detail is more important than minimum latency. These adapters were trained for the **FL2VA** partition and apply to `t2va` or `fl2va` requests. Do not use them with the separate `ref2va` weights unless the adapter author explicitly provides Ref2VA-compatible weights. Also avoid stacking a distilled adapter with `quality: "high"`: both alter denoising, and that combination has not been quality-validated. LoRAs trained for a pruned or structurally modified ComfyUI graph are not automatically compatible with the native H3 weights. Use only adapters whose architecture and target modules match the full native H3 checkpoint. ## 6. Sampling and output controls MiniMax-H3 supports more than one output per prompt. The video API accepts `num_outputs_per_prompt` (or OpenAI-compatible `n`) from 1 through 10. Offline generation accepts `--num-outputs-per-prompt N`; `--num-outputs N` is the short alias. A scalar seed is expanded deterministically as `seed + output_index`, so the outputs do not reuse the same noise. Same-prompt fan-out reuses text conditioning. On the verified 2× RTX 5090 recipe, a 5-step two-output request completed in 155.39 seconds versus 78.11 seconds for one output, while producing two distinct valid MP4 files. The independent denoise and decode passes remain sequential on this 32 GB profile to keep peak memory bounded; the grouped path adds essentially no orchestration overhead. Use server replicas when lower wall-clock latency for many variants matters more than per-server memory efficiency. For example, set `"num_outputs_per_prompt": 2` in any request above. After the job completes, download both outputs by selecting each zero-based variant: ```bash Command theme={null} video_id="" for variant in 0 1; do curl -sS -L \ "http://127.0.0.1:30010/v1/videos/${video_id}/content?variant=${variant}" \ -o "minimax-h3-${variant}.mp4" done ``` ### Choose the quality level `quality` is a request-scoped sampling parameter with two validated levels: * `"lossless"` (default): the exact reference path. Output is bit-exact against the reference implementation and the CI ground truth. * `"high"`: the audited accelerated path. Quality is guaranteed (the audited Cache-DiT configuration measures SSIM 0.931 / PSNR 28.16 dB against `lossless`), but output is no longer bit-identical to the reference. One resident server serves both levels; a `quality: "high"` request mounts its audited Cache-DiT policy at the batch boundary, and a later `quality: "lossless"` request removes the hooks before denoising. Start the validated server once: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 4 \ --tp-size 1 \ --sp-degree 4 \ --ulysses-degree 4 \ --ring-degree 1 \ --encoder-parallel auto \ --performance-mode speed \ --use-fsdp-inference false \ --enable-torch-compile false \ --port 30010 ``` Then choose a request level: Native denoising with no feature-cache approximation. This is the default; omitting the field is equivalent. ```json Request field theme={null} { "quality": "lossless" } ``` The audited accelerated path. Use it when you can trade bit-exactness for latency while keeping output closest to the same-seed lossless trajectory. ```json Request field theme={null} { "quality": "high" } ``` The measured trade-off is: | `quality` | Mean
inference
latency | Speedup | SSIM vs
lossless | PSNR vs
lossless | Expected
trade-off | | ---------- | ---------------------------------: | ------: | ---------------------: | ---------------------: | -------------------------------- | | `lossless` | 75.10 s | 1.00× | 1.000 | exact | Native reference path | | `high` | 53.70 s | 1.40× | 0.931 | 28.16 dB | Smallest same-seed visual change | These numbers use 1344×768, 124-frame, 24 fps T2VA with 50 inference steps, video flow shift 12, audio flow shift 3, and three fixed prompt/seed pairs on 4×H200. The prompts cover a quiet detailed scene, fast multi-subject action, and a moving close-up portrait. `inference_time_s` is averaged across the three prompts; the quiet-scene point is itself the mean of two repeats. SSIM and PSNR compare decoded, frame-aligned output with the `lossless` result for the same prompt and seed. They measure trajectory deviation, not absolute perceptual quality: the `high` path can produce a different but still plausible realization. It also changes the joint audio-video denoise trajectory, while these two metrics cover video only. `quality: "high"` currently accepts only the exact workload and 4×H200 deployment above; other hardware, task modes, request shapes, step counts, or flow shifts fail before denoising. Offline generation uses the same level name, for example `sglang generate --quality high`. `quality` selects a model sampling level and can change generated content. `output_quality` controls only output-file compression; it is a separate field. For manually tuned Cache-DiT experiments outside that validated path, omit the request `quality` field and set the process-wide environment controls directly. An explicit `quality: "lossless"` request overrides those controls and restores native denoising: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=1 \ SGLANG_CACHE_DIT_BN=0 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.12 \ SGLANG_CACHE_DIT_MC=2 \ sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --port 30010 ``` Cache-DiT skips selected block computation and is approximate. It cannot be combined with FSDP inference or DiT layerwise offload. Breakable CUDA graph execution takes precedence and leaves Cache-DiT disabled. Tune the cache thresholds only after comparing both video and audio quality on the target task profile. A real B200 request has completed, but the `quality: "high"` path above remains fail-closed to the audited 4×H200 workload. ## 7. Feature contracts and advanced recipes The generated command already contains the recommended topology and encoder setting. Use the detailed reference below only when applying an optional override or checking its installation, topology limits, and validation evidence. The recommended `speed` launch already combines resident components with Ulysses sequence parallelism. Validation status below applies only to the listed hardware and topology; it is not inherited by a similar GPU family. | Feature | Validation status | Notes | | -------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`. Combine with Ring for cross-node scaling; see the next row. | | Ring sequence parallelism (cross-node) | Verified: 2 nodes of 8× H200 each (Ulysses8 × Ring2) | Use `--ring-degree` together with `--nnodes`/`--node-rank`/`--dist-init-addr`. Ring shards the sequence across nodes while Ulysses shards heads within a node; H3's packed multi-segment attention only supports Ring across the node boundary, not within a single node's Ulysses group. Requires `--encoder-parallel replicate` — `auto`'s fold decision is not node-boundary aware. See the benchmark section below. | | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. | | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. | | Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. | | CPU and layerwise offload | Verified: 2× RTX 5090 TP2; 1× RTX 4090 24 GB | The 5090 lossless recipe keeps 20 DiT blocks plus both VAE encoders resident, streams the remaining DiT blocks, text encoder, and video VAE decoder blocks, and leaves the small audio VAE resident. The 4090 recipe streams DiT and the text encoder with zero resident DiT layers and **omits `vae`** from `--layerwise-offload-components`. | | Breakable CUDA graph | Verified: B200 Ref2VA, opt-in | Matching eager output was observed for the captured signature, without a measured speedup. Re-capture for other shapes and reference sets. | | `torch.compile` | Measured: H200, opt-in | Steady-state benefit was below measurement noise, while startup increased and numerical output changed. Do not use it for consistency ground truth. | The verified parallel, placement, and matching-signature BCG paths keep the BF16/FP32 weights and denoising math. `torch.compile` is the exception called out above. Always use the eager BF16/FP32 launch when producing CI consistency ground truth. For the validated 1344×768 Ref2VA profile, use a 5504-row text bucket so both the server warmup and reference-conditioned requests share the captured signature: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --enable-breakable-cuda-graph true \ --warmup-resolutions 1344x768 \ --bcg-text-buckets 5504 \ --port 30010 ``` BCG is lossless for a matching captured signature, but capture reserves extra GPU memory. Re-measure the live H3 text length before reusing this bucket for a different task profile, reference set, resolution, or prompt template. Leave `--attention-backend` unset for the platform default. Use `--attention-backend fa` only for an explicit FlashAttention comparison. SageAttention uses quantized attention math and is not a consistency mode. To select H3's native packed-varlen Sage path, install the dependency and add `--attention-backend sage_attn`. On Hopper, install the upstream SM90 binding fix rather than the PyPI 2.2.0 build: ```bash Command theme={null} pip install --force-reinstall \ git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 \ --no-build-isolation ``` The backend is a server-wide default. Use `--component-attention-backends` only when a measured component needs a different kernel, and keep the platform default for every component not named in the override. On the verified 8× B200 topology, quantize the BF16 transformer at server load: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --performance-mode speed \ --quantization fp8 \ --port 30010 ``` H3 automatically keeps its video/audio patch projections, timestep MLP, and final video/audio heads in FP32. All other linear layers have stable full module prefixes, so additional layers can be kept unquantized: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 8 \ --ulysses-degree 8 \ --encoder-parallel auto \ --quantization fp8 \ --quantization-ignored-layers blocks.0.attn token_refiner \ --port 30010 ``` Online FP8 is approximate and is not a consistency ground-truth mode. It can be combined with Cache-DiT, but the two approximations compound. Validate visual quality, audio quality, memory use, and latency on the target workload. This recipe is limited to the resident B200 and B300 topologies used for real H3 validation runs. On a single 24 GB card, use `kitchen_int8` instead of FP8. It quantizes the four GEMMs per DiT block online from the Hub BF16 weights (data-free, no calibration) and dispatches them through `comfy_kitchen.int8_linear`. Quantization happens after H3's grouped `qkv` reorder, so do not load an externally pre-quantized INT8 checkpoint here. ```bash Command theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --quantization kitchen_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` `fa` keeps exact attention. For a faster, approximate DiT path, use `--attention-backend sol_attn` with `--attention-backend-config dense_backend=sage_attn,dense_steps=10` and `--component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn`. See [Quantization](/docs/sglang-diffusion/quantization#kitchen-int8) and [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid). `kitchen_int8` changes Linear numerics. `sol_attn` / `sage_attn` also change the attention algorithm. Neither is a consistency ground-truth mode. The BF16 path is unchanged when `comfy-kitchen` is not installed. The Qwen3-VL text encoder can be replaced independently of the DiT. To reduce its resident memory, point the text-encoder component at the serialized FP8 checkpoint used in validation: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8 \ --num-gpus 4 \ --tp-size 2 \ --ulysses-degree 2 \ --performance-mode speed \ --port 30010 ``` `--text-encoder-path` is accepted as a shorter alias. No separate quantization flag is required: SGLang reads the checkpoint's `quantization_config` and fails closed if the native encoder does not support that format. The language linear layers use FP8 while embeddings, normalization, and the vision tower remain BF16. This is an approximate serve-time choice and is incompatible with the strict `quality="high"` deployment contract. The mixed W4A8 community encoder follows the same component-path contract: ```bash Overlay theme={null} --component-paths.text_encoder \ Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI/qwen3vl_32b_minimax_h3-w4a8_convrot.safetensors ``` Install `comfy-kitchen>=0.2.27` and omit `--quantization`. SGLang automatically loads its W4A8 language linears and tensorwise INT8 embedding; the unmarked vision tower remains BF16. W4A4 Qwen3-VL files use the same overlay, for example: ```bash Overlay theme={null} --component-paths.text_encoder \ Merserk/MiniMax-H3-INT4-ConvRot/qwen3vl_32b_minimax_h3_int4_convrot.safetensors ``` This checkpoint keeps its unmarked embedding and vision tower in their source precision; no explicit component quantization option is needed. The same component option accepts a self-describing Quanto qint8 file without an additional quantization flag: ```bash Overlay theme={null} --component-paths.text_encoder \ DeepBeepMeep/MiniMax-H3/Qwen3-VL-32B-Instruct/Qwen3-VL-32B-Instruct-layer50_quanto_bf16_int8.safetensors ``` This variant keeps the declared language and vision linear weights in qint8 storage, then dequantizes only the active matrix for BF16/FP16 linear math. Use it as a memory option, not as an INT8 throughput claim. The embedded Quanto map is the selector; adding `--quantization` would describe a different operation. Qwen3-VL GGUF files use the same component option: ```bash Overlay theme={null} --component-paths.text_encoder \ DeepBeepMeep/MiniMax-H3/Qwen3-VL-32B-Instruct/qwen3vl-32B-MiniMax-H3-Q4_K_M.gguf ``` The file selects its own Q4/Q6 layout; do not add `--quantization`. SGLang keeps the aligned vocabulary table and language linears in packed GGML storage. The smaller Comfy vision tower, whose flattened blocks can cross row and TP shard boundaries, is restored one matrix at a time to BF16 during loading. This CUDA-only memory option supports encoder TP and layerwise placement, but not encoder FSDP. Like the Quanto variant, it is approximate and is not a throughput claim. The picker explicitly writes `--encoder-parallel auto` in every single-node recipe. At the default request batch size of one, H100/H200/B200/B300 servers with peer-to-peer access fold the Qwen encoder across otherwise idle Ulysses ranks. A pure-TP recipe keeps the encoder inside its TP group, while a PCIe-only host can avoid an expensive world fold. Keep `auto` unless one of the cases below applies. Encoder DP is a throughput policy for compatible request batches. It requires TP1 and DiT DP1, replicates the encoder weights, and does not improve a batch of one: ```bash Overlay theme={null} --encoder-parallel dp \ --batching-max-size 2 ``` The cross-node picker recipe already uses replication because the automatic fold decision is not node-boundary aware: ```bash Overlay theme={null} --encoder-parallel replicate ``` ## 8. Configuration notes * MiniMax-H3 produces the canonical 24 fps output; request duration is expressed through `target.duration_seconds`. * `target.duration_seconds` must be between 4 and 15 seconds, inclusive. The command picker defaults to the verified 5-second profile. * Use a 768-pixel short edge for the released quality recipe. The aligned output dimensions are derived from `target.aspect_ratio`. * `flow_shift` controls video diffusion and `audio_flow_shift` controls audio diffusion. * V2V uses `task: "ref2va"` with a `video` or `video_audio` reference; it is served by the `Ref2VA` partition and is not a separate public task value. * `conditions[].start_time_seconds` selects a non-negative offset for a video reference. Its visual and audio streams are always sought together. * Ref2VA condition order is semantic and must match the one-based material tags in the prompt. For Ref2VA, `target.aspect_ratio: "auto"` resolves to the model's 16:9 fallback rather than inheriting a reference asset's geometry. * The distilled pipeline uses a single denoising branch, so CFG parallelism does not apply. Do not enable it: `--enable-cfg-parallel true` or `--cfg-parallel-size` greater than 1 is rejected instead of duplicating the positive branch. Explicitly disabling CFG, or setting its size to 1, remains a valid no-op. * The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile. * H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe. * Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold. * For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1. Encoder DP stays inside each DiT replica and composes with encoder TP: the H100 TP2 + Ulysses2 recipe has two TP-sharded encoder copies that can split a batch, while the RTX 5090 pure-TP2 recipe has one encoder copy and therefore no additional batch-DP degree. It provides no benefit for a batch of one and is not bitwise-identical to the unsplit deployment. * Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable. * `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster. * `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits. * Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it. ## 9. Benchmarks The picker exposes resident and FSDP profiles on NVIDIA datacenter GPUs. GPU counts are properties of the selected recipes, not a claim that every platform requires that many GPUs. The detailed tables below report performance only for the configurations with collected measurements: | Hardware | Default resident recipe | Other profile or topology | | --------------- | ------------------------------------- | ----------------------------------------------------------------------------- | | B300 | 8× Ulysses8 resident | 8× FSDP + Ulysses8; the 8-GPU sweep is not a minimum-GPU claim. | | B200 | 8× Ulysses8 resident | 4× FSDP + Ulysses4 | | H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2; 2 nodes × 8× Ulysses8×Ring2 cross-node | | H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 | | MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs | | RTX 5090 | 2× TP2 + layerwise offload | — | | RTX 4090 24 GB | 1× layerwise offload + `kitchen_int8` | Approximate attention backends are opt-in | ### B300 precision and encoder placement A 12-configuration sweep on a single 8× B300 host, covering both checkpoint partitions, both transformer precisions, and all three text-encoder placements. It answers one question — *how long does one request take, and how much memory does it need*. ### What was measured **Hardware.** 8× NVIDIA B300 SXM6, single node. **Model.** `MiniMaxAI/MiniMax-H3`, both released weight partitions. **Serve command.** Exactly the recipe the picker emits for B300, plus the one or two overlay flags under test: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --num-gpus 8 \ --ulysses-degree 8 \ --performance-mode speed \ --host 0.0.0.0 \ --port 30010 ``` The swept axes are `--model-variant` (`fl2va` / `ref2va`), `--quantization` (unset for BF16 / `fp8`), and `--encoder-parallel` (`auto` / `fold` / `replicate`). Nothing else differs between the 12 servers. This is a single-request latency sweep (`batching_max_size: 1`), so encoder DP is intentionally excluded: it cannot distribute a batch of one. Use the **DP for a request batch** setting above for a compatible multi-request deployment; the table below does not claim a measured H3 DP speedup. **Driver.** ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --host 127.0.0.1 --port 30010 \ --model MiniMaxAI/MiniMax-H3 \ --dataset vbench --task text-to-video \ --num-prompts 1 --max-concurrency 1 \ --warmup-requests 1 --warmup-inference-steps 50 \ --extra-body '{"task":"t2va","conditions":[],"target":{"short_edge":768,"aspect_ratio":"16:9","duration_seconds":5.0},"seconds":5,"flow_shift":12.0,"audio_flow_shift":3.0}' ``` **Workload** | Property | Value | | --------------------------------- | ----------------------------------------------------------- | | Output duration | 5.167 s | | Resolution | 1344×768 | | Frames | 124 @ 24 fps | | Denoising steps | 50 | | `flow_shift` / `audio_flow_shift` | 12.0 / 3.0 | | Requests in flight | 1 (`--max-concurrency 1`, server at `batching_max_size: 1`) | | Requests measured | 1 per cell, after 1 warmup request | ### Results | Weights | Precision | Encoder | Load | Warmup | Latency | Peak/GPU | | ------- | --------- | --------- | ------: | ------: | ----------: | ---------: | | FL2VA | BF16 | auto | 118.1 s | 29.65 s | **19.04 s** | 83,578 MB | | FL2VA | BF16 | fold | 114.0 s | 28.72 s | **19.04 s** | 83,578 MB | | FL2VA | BF16 | replicate | 116.0 s | 28.33 s | **19.04 s** | 124,158 MB | | FL2VA | FP8 | auto | 116.0 s | 27.16 s | **18.03 s** | 51,926 MB | | FL2VA | FP8 | fold | 116.0 s | 25.99 s | **18.04 s** | 51,926 MB | | FL2VA | FP8 | replicate | 118.0 s | 27.97 s | **18.04 s** | 92,506 MB | | Ref2VA | BF16 | auto | 114.0 s | 38.69 s | **29.12 s** | 83,968 MB | | Ref2VA | BF16 | fold | 118.0 s | 36.58 s | **29.13 s** | 83,968 MB | | Ref2VA | BF16 | replicate | 116.0 s | 35.17 s | **29.13 s** | 124,490 MB | | Ref2VA | FP8 | auto | 124.0 s | 34.30 s | **27.12 s** | 52,816 MB | | Ref2VA | FP8 | fold | 112.0 s | 34.44 s | **27.12 s** | 52,816 MB | | Ref2VA | FP8 | replicate | 116.0 s | 33.42 s | **27.12 s** | 93,396 MB | ### H200 topology comparison The same four-card H200 host completed both lossless resident placements with the standard 1344×768, 5-second, 50-step T2VA request (fixed prompt and seed, eager BF16/FP32, back-to-back runs on an otherwise idle host). Latency is the warmed-up request; the first pair uses the default warmup request, the second pair adds `--warmup-resolutions 1344x768` so warmup already covers the served resolution: | Topology | Warmup | Denoise | Decode | E2E | Peak/GPU | | -------------- | ------------------------------- | ------: | -----: | ----------: | --------: | | Ulysses4 | default | 79.04 s | 3.77 s | **84.14 s** | 94,288 MB | | TP2 + Ulysses2 | default | 81.17 s | 2.97 s | 85.51 s | 63,490 MB | | Ulysses4 | `--warmup-resolutions 1344x768` | 71.73 s | 1.32 s | **74.38 s** | 94,290 MB | | TP2 + Ulysses2 | `--warmup-resolutions 1344x768` | 75.52 s | 1.29 s | 78.33 s | 63,490 MB | Ulysses4 stays the H200 latency default: 5.0 % faster end-to-end than TP2 + Ulysses2 once warmup covers the served resolution (1.6 % with the default warmup, where first-request cold start masks the topology gap). TP2 + Ulysses2 shards the DiT weights and holds peak memory about 30 GB per GPU lower, which is why it remains the 80 GB H100 recipe. Matching the warmup request to the served resolution removes the cold first-request cost on both topologies (about 10 s end-to-end on this workload). ### H200 cross-node scaling Long references and long durations grow the packed sequence length, and Ulysses alone cannot scale sequence parallelism past the GPU count of one node without either violating head-count divisibility or exposing all-to-all traffic across the slower inter-node link. H3 combines node-local Ulysses with cross-node Ring: Ring's point-to-point KV rotation is designed to overlap with attention compute, which fits a slower cross-node link better than an all-to-all does. **Hardware.** 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand between nodes. **Serve command.** The cross-node cell the picker emits for H200, run identically on both nodes with `--node-rank` set to 0 and 1: ```bash Command theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 16 \ --nnodes 2 \ --node-rank {{NODE_RANK}} \ --dist-init-addr {{NODE0_IP}}:20000 \ --sp-degree 16 \ --ulysses-degree 8 \ --ring-degree 2 \ --encoder-parallel replicate \ --performance-mode speed \ --host 0.0.0.0 \ --port 30010 ``` **What was measured.** A controlled denoise-stage comparison on identical hardware: 8× H200 single-node (Ulysses8, no Ring) versus the same 16-GPU cross-node command above (Ulysses8 × Ring2), holding prompt, seed, and step count fixed: | Task | Single-node (Ulysses8) | Cross-node (Ulysses8 × Ring2) | Change | | ----------------------- | ---------------------: | ----------------------------: | -----: | | T2VA denoise/step | 0.749 s | 0.477 s | −36.3% | | Ref2VA/V2V denoise/step | 2.572 s | 1.494 s | −41.9% | The gain grows with sequence length because Ring's per-hop communication cost stays roughly constant while attention compute grows quadratically with sequence length, so V2V's longer packed sequence benefits more than T2VA's shorter one. With the point-to-point KV rotation pipelined against attention compute, one V2V request's full denoise stage completed in 68.1–68.3 seconds versus 128.6 seconds on the single-node 8-GPU baseline (−47.0%), with byte-identical output to the unpipelined cross-node path. Cross-node determinism was confirmed separately: the same request run twice against the same cross-node deployment produced byte-identical output. A cross-node run's output is not expected to bit-match a single-node run of the same prompt and seed — Ring's online-softmax merge across hops accumulates floating-point operations in a different order than single-node attention, which is an expected source of bit-level difference, not a correctness regression. `--encoder-parallel auto`'s fold decision is not yet node-boundary aware and attempts to fold the text encoder across nodes, which crashes the Ref2VA reference-conditioned encoder. Always pass `--encoder-parallel replicate` explicitly for cross-node H3 deployments. ### H100 topology comparison The same four-card H100 host completed three lossless placements. TP2 with Ulysses2 was the fastest; TP4 used the least memory: | Topology | Pipeline latency | Peak/GPU | | --------------- | ---------------: | -------: | | TP2 + Ulysses2 | 13.25 s | 66.04 GB | | FSDP + Ulysses4 | 13.36 s | 57.01 GB | | TP4 + Ulysses1 | 13.86 s | 49.80 GB | ### RTX 5090 capacity run The verified two-card RTX 5090 host used TP2 with layerwise offload. The full 50-step, 1344×768, 5-second request completed in 559.67 seconds: 525.05 seconds of denoising and 33.61 seconds of decoding, with a 26.3 GiB sampled peak per GPU. | DiT settings | 5-step denoise | Inference | Peak/GPU | Result | | --------------------------------- | -------------------: | --------: | -------: | ------------------ | | prefetch 1, resident 20 | 43.48 s | 78.11 s | 26.3 GiB | Selected recipe | | prefetch 2, resident 20 | 43.37 s | 78.06 s | 27.5 GiB | No measurable gain | | Ulysses2, prefetch 2, resident 10 | Did not reach warmup | — | — | Rejected | ### Consumer GPU tuning On consumer hardware the binding question is not which card you have but how much host RAM sits behind it. H3's weights are about 108 GB — 61.73 GB of DiT and 46.18 GB of text encoder — so no consumer configuration holds them all, and where the shortfall lands decides the throughput. **The command** — most consumer machines need exactly one flag beyond the model: ```bash consumer single GPU, lossless theme={null} sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --layerwise-offload-components dit,text_encoder,vae ``` With 16 GB of VRAM or more, add `--layerwise-resident-layers video_vae=36` for the 13 s decode; with \~96 GB of host RAM and 16 GB+ of VRAM, add `--dit-layerwise-resident-layers 4` for the 6 s step. That is the whole flag surface. The [builder at the top of this page](#1-quick-start) has consumer cards and a Host RAM selector: pick your budget and it emits this command with your tier's measured expectations attached as comments. The table below is the same data in one view. **Two budgets, and what each one buys** | | 12 GB VRAM + 32 GB host | host free, VRAM 16 GB | | -------------------------------------- | ----------------------: | --------------------: | | Recipe | A | B | | Peak VRAM | ≤ 12 GiB | ≤ 16 GiB (OOMs at 12) | | Host anonymous (must fit) | 24.5 GiB | 116.7 GB pinned | | Denoise, 864×480 / 124 frames / 20 NFE | 16.8 - 18.7 s/it | **6.01 s/it** | | Runs at all | yes | yes | The left column is one configuration measured twice, at 318.94 s and 356.37 s; the 12% spread tracked host load on a shared machine, so treat smaller differences than that as unresolved. The right column is 120.92 s at a 16 GiB allocator cap. Four resident DiT layers is what Recipe B buys its speed with, and it is also why 12 GiB is not enough for it. Read the host row carefully, because the two numbers are not the same kind of memory. *Anonymous* host memory — pinned buffers and pageable copies — has to fit, and the kernel cannot reclaim it. Page cache backing a file mapping is *droppable*, so it does not count against the budget even though it shows up in `VmRSS`; use `RssAnon` from `/proc//status` when checking. Likewise measure VRAM with `torch.cuda.set_per_process_memory_fraction` and let the allocator fail, rather than reading `nvidia-smi`, which reports the caching allocator's reserved pool and overstates the requirement. Inside 32 GB the weights cannot be pinned, so each denoise step copies about 60 GiB from the checkpoint mapping, and a mapped source is synchronous however the copy is requested: the driver stages it through its own buffer, so the transfer neither overlaps compute nor runs at pinned bandwidth. That is where the step goes, and giving the host room to pin the weights instead is what takes it to 6.01 s. Two caveats on the constrained number, both from instrumenting the run rather than from arithmetic. The machine it was measured on has 2 TB of host memory, so the kernel kept all 107.7 GiB of mapped checkpoint pages resident: major faults across a whole request were 6, and `read_bytes` was zero. Nothing was read from disk. A real 32 GB host cannot cache 107.7 GiB, so it will fault and re-read, and should be expected to be slower than the figures here rather than equal to them — an NVMe is a requirement, not a recommendation. Measure your own machine with major faults (`/proc//stat`) on the worker process, not on the launcher, which holds no weights. **Recipe A — fits 12 GB VRAM + 32 GB host** ```bash 12 GB + 32 GB, lossless theme={null} PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --layerwise-resident-layers video_vae=36 ``` **Recipe B — host memory is free (the fast path)** ```bash unconstrained host, lossless theme={null} sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder,vae \ --dit-layerwise-resident-layers 4 \ --layerwise-resident-layers video_vae=36 ``` Recipe B pins \~112 GB of host memory (DiT 61.56 GB, text encoder 46.18 GB, VAE \~4.5 GB in its decode dtype). Do not reach for it on a 32 GB machine. **What not to change, and why** * `video_vae=36` holds every decoder block for the decode only — residency arms at the decoder's first block and releases when it finishes, so the denoise still runs on an empty card. It fits 12 GB because decoder weights are held in their decode compute dtype (fp16) from load, which halves them to \~4.9 GiB; the rounding was already part of every output (the decode computes in fp16 autocast), so the result is bit-identical, and the decode drops from 60 s streamed (or 209 s on a busy host) to \~10 s. The `expandable_segments` line stays: the decode sits close enough to the cap that fragmentation otherwise tips it over. * Leave `--enable-torch-compile` off, as elsewhere on this page. Layerwise offload rebinds `param.data` on every layer, so compiled graphs do not get the benefit they would on resident weights. * Recipe A's flags are what the automatic policy should choose on its own. Until the model declares its own placement, `--performance-mode memory` plus the explicit component list is what makes it happen; pass them. **Reading the startup log** The server prints the memory decisions it made; checking three lines against your budget catches a mis-set machine in the first minute instead of the first request. * `Layerwise offload: host memory available: N GiB` — what the runtime sees after loading, not your DIMM size. On a 32 GB host expect single digits here; a much larger number means another process's memory accounting (or a container limit) is in play. * `leaving N GiB of weights on the checkpoint mapping` — the expected line on a 32 GB host: the DiT streams from the checkpoint file. If instead the log reports pinned weights, the runtime decided your host has room — which is faster, and means the 32 GB figures above do not apply to you. * `Loaded video_vae: ... host mmap` vs `host pageable` — where the VAE landed (decoder weights are \~4.9 GiB once held in their decode dtype). `Loaded ` lines carry the same buckets for every component. If a request dies after the denoise finishes, it is the decode colliding with the cap: keep the `expandable_segments` line, and if it persists drop to `video_vae=24` and take the partially streamed decode. **Against ComfyUI, on the same weights** Same unpruned bf16 checkpoints, same card, same sampler settings (cfg 1.0, euler\_ancestral, sigma shift 12.0/3.0, seed 1101), 864×480 / 124 frames / 20 NFE: When host memory is free, the engines are close and sglang is ahead: | | denoise | host anonymous | peak VRAM | | ---------------- | -------------: | --------------: | --------: | | sglang, Recipe B | **6.01 s/it** | 116.7 GB pinned | ≤ 16 GiB | | ComfyUI KSampler | 6.58–6.59 s/it | 116.5 GiB | 13048 MiB | Inside 12 GB, both engines run these weights, and one measurement convention matters on each side. ComfyUI's memory manager reads system RAM and adapts, so the rows below patch `psutil` to a pretend host size — the same convention the sglang rows use. Its `--reserve-vram` is also soft: told to keep 12 GiB free it still peaked at 13.5 GiB, a figure a real 12 GB card cannot give it, so both engines here run under the same hard allocator cap (`set_per_process_memory_fraction`), where its peak stays at 12.1–12.3 GiB. Under that cap, Recipe A wins the whole request at every host size: | 12 GB VRAM, both engines hard-capped | sglang Recipe A (TE + denoise + decode) | ComfyUI, bf16 (warm) | | ------------------------------------ | --------------------------------------: | -------------------: | | 32 GB host | 12.4 + 212.8 + 9.4 ≈ **235 s** | 276–302 s | | 48 GB host | 15.8 + 192.1 + 10.0 ≈ **218 s** | 246–267 s | | 64 GB host | 7.5 + 162.4 + 10.3 ≈ **180 s** | 194–195 s | Same GPU, same load window, unpruned bf16 checkpoints, outputs verified. The VRAM axis holds too: capped at 16 GiB the same recipe wins \~250 vs 292–301 s, and at 24 GiB (with `--dit-layerwise-resident-layers 10`, which only a 24 GB card has headroom for) \~230 vs 249–260 s. Four changes carry it: the VAE staying on its checkpoint mapping (#35862, root fix \#35946), per-layer pinning with net-cost accounting (#35867), the courier thread that ships still-mapped layers through pinned slots (#35882), and decoder weights held in their decode dtype from load (#35967) — which is what lets `video_vae=36` fit and turns the decode from the slowest stage (54–96 s streamed) into the fastest (\~10 s, faster than ComfyUI's own 15–25 s). Output equivalence is bit-level: the fp16-held decode reproduced the fp32-store run's video byte for byte, and the audio stream is bit-identical. Stage by stage under the cap: text encoding is even (both stream the same 48 GB Qwen3VL), the denoise leads at 32–48 GB hosts and sits within run-to-run variance of ComfyUI at 64 GB (162 vs 159 s), and the decode leads everywhere. Two ComfyUI notes that still matter: `--fast-disk` measured no faster than its default here, and stacking `--novram --cache-none --disable-pinned-memory` made things strictly worse (69.1 GiB anonymous, 750 s requests) — the adaptive default is the right configuration on a small host. The path ComfyUI ships for 12 GB cards uses `minimax_h3_fl2va_pruned_int8_convrot` and `qwen3vl_32b_minimax_h3_nvfp4_awq`, i.e. an int8 DiT and an NVFP4 text encoder, and its pruned bf16 file is 40.2 GB against the unpruned 66.3 GB. Those are different weights, so it is not a like-for-like comparison with the recipes above. ### RTX 4090 24 GB single-GPU run One RTX 4090 D 24 GB completed the 1344×768, 107-frame, 20-NFE T2VA workload (euler, `torch.compile` and step caching disabled) with DiT and text-encoder layerwise offload. Same process: load → warmup (seed 0) → timed (seed 42); only the timed pass is reported. GPU peak stayed about 18 GB. | Config | Timed e2e | Denoise | vs BF16 | PSNR vs BF16 | | -------------------------------- | --------: | ------: | ------: | -----------: | | BF16 + FlashAttention | 405.6 s | 370.2 s | 1.00× | — | | `kitchen_int8` + FA | 303.3 s | 273.7 s | 1.34× | 24.81 dB | | `kitchen_int8` + `sol_attn` | 223.9 s | 203.9 s | 1.81× | 24.44 dB | | `kitchen_int8` + `sage_attn` | 174.9 s | 154.2 s | 2.32× | 23.51 dB | | `kitchen_int8` + Sage→Sol hybrid | 163.8 s | 143.1 s | 2.48× | 23.04 dB | `kitchen_int8` + FA changes Linear numerics only. The `sol_attn` / `sage_attn` / hybrid rows also change the attention algorithm, so speed and pixel fidelity rank in opposite orders there. Default remains `kitchen_int8` + `fa`. ### AMD Instinct task and scaling runs The AMD recipes keep the released BF16/FP32 precision policy and use AITER packed attention. The picker emits the fastest measured topology, 8 GPUs with Ulysses degree 8. All runs below completed full H.264/AAC decoding and representative-frame inspection. | Hardware | Task | Denoise | Decode | Peak/GPU | | -------- | ------ | ---------: | --------: | --------: | | MI355X | T2VA | 55.2907 s | 9.5344 s | 97,444 MB | | MI355X | FL2VA | 53.7978 s | 9.4477 s | 96,922 MB | | MI355X | Ref2VA | 41.3812 s | 6.8247 s | 94,518 MB | | MI300X | T2VA | 167.4878 s | 25.3244 s | 97,272 MB | | MI300X | FL2VA | 150.2311 s | 12.5684 s | 96,750 MB | | MI300X | Ref2VA | 107.6232 s | 11.3768 s | 94,268 MB | The task matrix used 8 GPUs and 50 denoising steps. The scaling matrix uses one 1344×768, 209-frame T2VA request and changes only the GPU count and matching Ulysses degree: | Hardware | GPUs | Denoise | Decode | Peak/GPU | | -------- | ---: | ---------: | --------: | ---------: | | MI355X | 8 | 55.2907 s | 9.5344 s | 97,444 MB | | MI355X | 4 | 104.2294 s | 11.1824 s | 103,350 MB | | MI355X | 2 | 223.0246 s | 15.5330 s | 115,250 MB | | MI355X | 1 | 288.7968 s | 24.0472 s | 137,676 MB | | MI300X | 8 | 167.4878 s | 25.3244 s | 97,272 MB | | MI300X | 4 | 297.3727 s | 26.5067 s | 103,436 MB | | MI300X | 2 | 585.5401 s | 29.4909 s | 115,010 MB | | MI300X | 1 | 978.0886 s | 36.0142 s | 137,626 MB | For a measured lower-count AMD deployment, set both `--num-gpus` and `--ulysses-degree` to 4, 2, or 1. AITER packed attention matched segment-wise BF16 SDPA at cosine similarity `0.9999991655` on MI355X and `0.9999991059` on MI300X. # Qwen-Image Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image ## 1. Model Introduction [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a 20B text-to-image model built for strong prompt following and precise rendering of English and Chinese text. It is especially useful for posters, signs, diagrams, and dense layouts where typography and spatial relationships matter as much as general image quality. This page covers generation rather than editing: use Qwen-Image-Edit when an existing image, subject identity, or untouched region must be preserved. The full checkpoint is memory-heavy, while the validated ModelOpt NVFP4 release provides a supported low-precision deployment option with an expected quality tradeoff. ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. For the validated ModelOpt NVFP4 checkpoint on Blackwell, load the published Qwen-Image-2512 NVFP4 repo directly: ```bash Command theme={null} sglang serve \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --ulysses-degree=1 \ --ring-degree=1 ``` For high-resolution B200 generations, the FlashInfer CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM FP4 GEMM backend: ```bash Command theme={null} SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \ sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --width 2048 --height 2048 \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP **AMD ROCm Notes**: Requires SGLang >= v0.5.8. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="Qwen/Qwen-Image", prompt="A logo With Bold Large text: SGL Diffusion", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Qwen/Qwen-Image ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark Test Environment: * Hardware: AMD Instinct MI300X GPU (1x) * Model: Qwen/Qwen-Image * Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x * sglang diffusion version: 0.5.8 ### 5.1 Speedup Benchmark #### 5.1.1 Generate an image **Server Command**: ```shell Command theme={null} sglang serve --model-path Qwen/Qwen-Image \ --ulysses-degree=1 --ring-degree=1 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 29.04 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.03 Latency Mean (s): 29.0378 Latency Median (s): 29.0378 Latency P99 (s): 29.0378 -------------------------------------------------- Peak Memory Max (MB): 48018.83 Peak Memory Mean (MB): 48018.83 Peak Memory Median (MB): 48018.83 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 36.26 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.03 Output throughput (outputs/s): 0.03 Latency Mean (s): 36.26 Latency Median (s): 36.26 Latency P90 (s): 36.26 Latency P95 (s): 36.26 Latency P99 (s): 36.26 -------------------------------------------------- Peak Memory Max (MB): 36984.00 Peak Memory Mean (MB): 36984.00 Peak Memory Median (MB): 36984.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.79 Request rate: inf Max request concurrency: 20 Successful requests: 14/20 -------------------------------------------------- Request throughput (req/s): 0.05 Latency Mean (s): 154.5368 Latency Median (s): 154.8363 Latency P99 (s): 285.4603 -------------------------------------------------- Peak Memory Max (MB): 48030.31 Peak Memory Mean (MB): 48030.30 Peak Memory Median (MB): 48030.29 ============================================================ ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Qwen/Qwen-Image Dataset: vbench -------------------------------------------------- Benchmark duration (s): 300.81 Request rate: inf Max request concurrency: 20 Successful requests: 8/20 Completed outputs: 8 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.03 Output throughput (outputs/s): 0.03 Latency Mean (s): 166.61 Latency Median (s): 167.02 Latency P90 (s): 270.80 Latency P95 (s): 283.48 Latency P99 (s): 293.64 -------------------------------------------------- Peak Memory Max (MB): 36984.00 Peak Memory Mean (MB): 36984.00 Peak Memory Median (MB): 36984.00 ------------------------------------------------------------ ``` # Qwen-Image-Edit-2511 Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit ## 1. Model Introduction [Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is the 20B editing counterpart to Qwen-Image. It is strongest at changing text, materials, lighting, viewpoint, or composition while reducing drift in regions that were not requested to change. Choose it for identity-sensitive portrait edits, multi-person composition, typography replacement, and geometry-aware design work. It is substantially heavier than small specialist editors, and consistency is improved rather than guaranteed; evaluate untouched-region drift on the actual editing workload. ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. The recommended launch configurations vary by hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Edit an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1") response = client.images.edit( model="Qwen/Qwen-Image-Edit-2511", image=open("input.png", "rb"), prompt="Change the color of the taxi to black.", n=1, response_format="b64_json", ) # Save the edited image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image-Edit-2511 ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Qwen/Qwen-Image-Edit-2511 ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: Qwen/Qwen-Image-Edit-2511 * sglang diffusion version: 0.5.6.post2 ### 5.1 Speedup Benchmark #### 5.1.1 Edit a image **Server Command**: ```shell Command theme={null} sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench Task: image-to-image -------------------------------------------------- Benchmark duration (s): 35.31 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.03 Latency Mean (s): 35.3053 Latency Median (s): 35.3053 Latency P99 (s): 35.3053 -------------------------------------------------- Peak Memory Max (MB): 47959.35 Peak Memory Mean (MB): 47959.35 Peak Memory Median (MB): 47959.35 ============================================================ ``` #### 5.1.2 Edit a image with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Qwen/Qwen-Image-Edit-2511 Dataset: vbench Task: image-to-image -------------------------------------------------- Benchmark duration (s): 286.11 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.07 Latency Mean (s): 150.0428 Latency Median (s): 150.0600 Latency P99 (s): 283.3843 -------------------------------------------------- Peak Memory Max (MB): 47971.82 Peak Memory Mean (MB): 47971.49 Peak Memory Median (MB): 47971.29 ============================================================ ``` # SANA-Video Source: https://docs.sglang.io/cookbook/diffusion/SANA-Video/SANA-Video Serve the native SANA-Video 2B 480p text-to-video model with SGLang Diffusion. ## 1. Model Introduction [SANA-Video 2B 480p](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers) is a lightweight text-to-video model served through a native SGLang Diffusion pipeline. Its main advantage is deployment cost: at 2B parameters it serves 480p video from a single GPU with a plain `sglang serve` invocation, no parallelism flags required. The released generation profile produces 832×480 output at 81 frames and 16 FPS over 50 inference steps; a compact 17-frame, 8-step profile is covered by server CI for quick validation. Motion strength can be steered directly from the prompt with an optional `motion score: N.` suffix. | Model ID | Task | Default output | | ---------------------------------------------------- | ------------- | ---------------------------- | | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | Text to video | 832x480, 81 frames at 16 FPS | ## 2. Installation Install SGLang with the diffusion dependencies: ```bash Command theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` See the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for platform-specific setup. ## 3. Serve SANA-Video ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \ --port 30010 ``` ## 4. Generate a video The following request uses the compact 17-frame, 8-step profile covered by server CI. Use the model defaults of 81 frames and 50 steps for the released generation profile. ```python Python theme={null} import time from pathlib import Path import requests base_url = "http://127.0.0.1:30010" response = requests.post( f"{base_url}/v1/videos", json={ "model": "Efficient-Large-Model/SANA-Video_2B_480p_diffusers", "prompt": ( "A red tram moves slowly through a sunlit city square while " "pedestrians cross behind it. motion score: 30." ), "size": "832x480", "num_frames": 17, "fps": 16, "num_inference_steps": 8, "guidance_scale": 6.0, "seed": 42, }, timeout=60, ) response.raise_for_status() video_id = response.json()["id"] while True: job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json() if job["status"] == "completed": break if job["status"] == "failed": raise RuntimeError(job.get("error") or "Video generation failed") time.sleep(1) video = requests.get( f"{base_url}/v1/videos/{video_id}/content", timeout=300, ) video.raise_for_status() Path("sana_video.mp4").write_bytes(video.content) ``` ## 5. Request constraints * The default profile uses `832x480`, 81 frames, 50 inference steps, and 16 FPS. * Frame counts are aligned to `4n+1`; for example, a request for 80 frames is adjusted to 77. * Use width and height values divisible by 16. * The prompt supports an optional `motion score: N.` suffix to express the desired amount of motion. # SANA-WM Source: https://docs.sglang.io/cookbook/diffusion/SANA-WM/SANA-WM ## 1. Model Introduction [SANA-WM](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) is NVLabs' 2.6B text-and-image-to-video world model for 720p, minute-scale generation with explicit per-frame 6-DoF camera control. Its hybrid recurrent/softmax attention keeps long causal histories bounded, while an LTX-2 refiner supplies the detail that the fast Stage-1 world model does not produce on its own. Choose the dense checkpoint for the best bounded-clip quality and the streaming checkpoint for long-running or interactive control. Streaming and realtime trade global bidirectional context for bounded state and lower response latency; the realtime WebSocket path is not bit-identical to offline batch streaming. | Serving mode | Checkpoint and API | Best fit | Tradeoff | | ------------------- | ---------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------- | | Dense bidirectional | `SANA-WM_bidirectional` via `/v1/videos` | Highest single-clip quality | Bounded one-shot generation; full attention cost | | Batch streaming | `SANA-WM_streaming` via `/v1/videos` | Long clips returned as one job | Chunk-causal context rather than global bidirectional context | | Live realtime | `SANA-WM_streaming` via WebSocket | Interactive camera events and incremental output | Stateful session; not bit-identical to batch streaming | **Architecture & components** | Component | Value | | ----------- | ------------------------------------------------------------------- | | Stage-1 DiT | 2.6B; 20 layers, hidden 2240, 20 heads (head\_dim 112); \~10 GB | | Attention | frame-wise Gated DeltaNet + softmax every 4th block (hybrid linear) | | Camera | dual-branch, UCPE + PRoPE (raymap + Plücker), 6-DoF | | VAE | LTX-2 causal, strides (T, H, W) = (8, 32, 32); \~2 GB | | Refiner | LTX-2 Stage-2 distilled; \~41 GB | | Output | up to 720p (704×1280) @ 16 fps, minute-scale | For architecture and training details, see the [SANA-WM paper](https://arxiv.org/abs/2605.15178) and [model card](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) (Apache-2.0). ## 2. Installation SGLang-diffusion offers multiple installation methods depending on your hardware platform. Please refer to the [SGLang Diffusion installation guide](../../../docs/sglang-diffusion/installation). SANA-WM adds the `SanaWMTransformer3DModel` + GDN kernels, the `SanaWMTwoStagePipeline` (dense bidirectional + chunk-causal streaming), and the `SanaWMRealtimePipeline` with the `/v1/realtime_video` WebSocket router. Use `sglang serve` to launch the diffusion server. ## 3. Model Setup Both SANA-WM checkpoints are **public** (Apache-2.0, no gating, no token) and load **directly** — there is no manual assembly step. Pass the HuggingFace repo id to `--model-path` and SGLang downloads, materializes, validates, and loads it: | Mode | `--model-path` | | ------------------------------------ | --------------------------------------------- | | Dense bidirectional (§4) | `Efficient-Large-Model/SANA-WM_bidirectional` | | Batch streaming (§5) / realtime (§6) | `Efficient-Large-Model/SANA-WM_streaming` | Both repo ids are registered in SGLang's **built-in model-overlay registry**, so on first load the overlay transparently materializes the official release into a runnable Diffusers directory — for the streaming checkpoint this converts the DMD self-forcing checkpoint (`sana_dit/model.pt`) into a Diffusers `transformer/` and wires the LTX-2 causal VAE, the LTX-2 refiner, and the Gemma encoders. No environment variable or `build_model_dir.sh` step is needed. (You may also pass a local, already-materialized Diffusers directory.) The materialized checkpoint is a Diffusers directory whose `model_index.json` declares the loadable components: | Component (`model_index.json`) | Class | | ------------------------------ | ------------------------------------------- | | `transformer` (Stage-1 DiT) | `diffusers.SanaWMTransformer3DModel` | | `vae` | `diffusers.AutoencoderKLCausalLTX2Video` | | `text_encoder` | `transformers.Gemma2Model` | | `tokenizer` | `transformers.GemmaTokenizer` | | `scheduler` | `diffusers.FlowMatchEulerDiscreteScheduler` | How loading works: * The server resolves the checkpoint via `maybe_download_model(model_path, force_diffusers_model=True)` and verifies it contains a `model_index.json` plus the required component subdirectories (`transformer/`, `vae/`). * If `text_encoder` / `tokenizer` are not provided as component paths, the pipeline falls back to the default Stage-1 text encoder **`Efficient-Large-Model/gemma-2-2b-it`** (`DEFAULT_SANA_WM_TEXT_ENCODER`). * **Pick the path with `--pipeline-class-name`.** The checkpoint's `model_index.json` `_class_name` selects the default pipeline (`SanaWMTwoStagePipeline`). Pin it explicitly to choose: `--pipeline-class-name SanaWMTwoStagePipeline` for the `/v1/videos` paths (§4–5) or `--pipeline-class-name SanaWMRealtimePipeline` for live realtime (§6). Pinning is also required if you point `--model-path` at a bare safetensors file instead of a Diffusers directory. * **The Stage-2 LTX-2 refiner** lives under `refiner/` in the checkpoint: `refiner/transformer` (`transformer_2`), `refiner/connectors` (`connectors`), and `refiner/text_encoder` (the Gemma-3 encoder for `text_encoder_2`, whose tokenizer also serves as `tokenizer_2`). The refiner is **optional**: it is skipped (Stage-1-only output) when the env flag `SGLANG_SANA_WM_SKIP_REFINER` (or a `skip_refiner` request extra) is set, or when no `refiner/` is present (`transformer_2` unloaded). On the batch path it runs chunk-wise with `--refiner-chunked` (the official streaming path, default on) or whole-clip without it; on the realtime path the pipeline builds a `SanaWMChunkedRefinerChainStage` only when a refiner is available, and otherwise streams Stage-1 frames. Throughout this cookbook, `` stands for the appropriate SANA-WM repo id from the table above (or a local materialized Diffusers directory). ## 4. Dense bidirectional (offline `/v1/videos`) The **bidirectional** checkpoint generates the whole clip in **one shot** (full bidirectional attention, not chunked) followed by a dense LTX-2 refiner — the highest single-clip quality, matching the NVlabs dense reference. Launch with the two-stage pipeline and **no** `--streaming` flag (dense is the default — `streaming` defaults to `False`): ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_bidirectional \ --pipeline-class-name SanaWMTwoStagePipeline \ --host 127.0.0.1 --port 30000 ``` Then POST to **`/v1/videos`** exactly as in §5, but pass the NVlabs dense sampling defaults for closest parity — the dense path is denser than the distilled streaming few-step schedule: ```bash Command theme={null} curl -s http://127.0.0.1:30000/v1/videos \ -H 'content-type: application/json' -d '{ "prompt": "a camera moving forward and turning left", "input_reference": "/path/to/first_frame.png", "num_frames": 321, "seed": 42, "fps": 16, "num_inference_steps": 60, "guidance_scale": 5.0, "diffusers_kwargs": { "action": "w-80,wl-80,l-80,wj-80", "intrinsics": "/path/to/intrinsics.npy" } }' ``` * `num_inference_steps` / `guidance_scale` — the dense path uses CFG; NVlabs' reference defaults to **60 steps, guidance 5.0** (the `SanaWMSamplingParams` defaults are the lighter 20 / 4.5 — pass 60 / 5.0 explicitly for dense parity). * The dense refiner drops the leading sink frame, so a `num_frames=321` request yields 320 output frames. ## 5. Batch streaming (offline `/v1/videos`) The **streaming** checkpoint generates a **full camera-controlled clip in one request** — no websocket. This is SGLang's offline streaming path: the whole clip is generated chunk-by-chunk internally, refined, decoded, and returned as one video. Launch with the two-stage pipeline + the streaming flags: ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMTwoStagePipeline \ --streaming --refiner-chunked \ --host 127.0.0.1 --port 30000 ``` * `--streaming` — chunk-causal `forward_long` Stage-1 (vs the dense one-shot path of §4). * `--refiner-chunked` — chunk-wise streaming LTX-2 refiner (**on by default**). To use the whole-clip dense refiner instead (also valid, higher peak memory), pass `--refiner-chunked false` — simply omitting the flag keeps the default chunked refiner. * `--num-frame-per-block N` — latent frames per chunk (default `3`). Then POST to **`/v1/videos`** (JSON body shown below; multipart/form-data with an uploaded `input_reference` file also works). Camera control goes in `diffusers_kwargs` — the action-DSL string (§8) and the intrinsics: ```bash Command theme={null} curl -s http://127.0.0.1:30000/v1/videos \ -H 'content-type: application/json' -d '{ "prompt": "a camera moving forward and turning left", "input_reference": "/path/to/first_frame.png", "num_frames": 321, "seed": 42, "fps": 16, "diffusers_kwargs": { "action": "w-80,wl-80,l-80,wj-80", "intrinsics": "/path/to/intrinsics.npy" } }' ``` | Field | Notes | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `prompt` | text prompt | | `input_reference` | first-frame image — a server-side path, or (multipart) an uploaded file. For an `http(s)://` URL in a JSON body, use the separate `reference_url` field (the server downloads it and assigns it to `input_reference`) | | `num_frames` | total pixel frames (e.g. `321` → 41 latent frames, 13 chunks; output 704×1280) | | `seed` | RNG seed (default `42`) | | `fps` | output frame rate — **pass `16`** (SANA-WM's native rate). The generic `/v1/videos` default is `24`, which would encode the same frames at 24 fps and make the clip play \~33% shorter (16/24 of the duration) | | `diffusers_kwargs.action` | camera action-DSL string (§8) | | `diffusers_kwargs.intrinsics` | path to a camera-intrinsics `.npy` (per-frame `(T,3,3)`) or an inline 3×3 / `(T,3,3)` list | The response is a `VideoResponse`; fetch the rendered MP4 via the returned reference or `GET /v1/videos/{id}/content`. The streaming hyperparameters (`num_frame_per_block`, `denoising_step_list`, `sink_size`, `num_cached_blocks`, `streaming_cfg_scale`) are **pipeline-config** defaults on `SanaWMPipelineConfig`, not request fields — see §9. ## 6. Launch the Realtime Server Launch with the realtime pipeline **pinned** — the checkpoint defaults to `SanaWMTwoStagePipeline`, so realtime must be selected explicitly (see §3). The `/v1/realtime_video` router is always mounted and becomes functional once the realtime config is active, because `SanaWMRealtimeConfig` has a registered realtime adapter (`SanaWMRealtimeAdapter`). ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --host 127.0.0.1 --port 30000 ``` Common launch variants: ```bash Command theme={null} # recommended multi-GPU realtime profile sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --num-gpus 8 --sp-degree 8 \ --host 127.0.0.1 --port 30000 # single GPU sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --num-gpus 1 --host 127.0.0.1 --port 30000 # offload DiT + text encoder to CPU (tight VRAM) sglang serve \ --model-path Efficient-Large-Model/SANA-WM_streaming \ --pipeline-class-name SanaWMRealtimePipeline \ --host 127.0.0.1 --port 30000 \ --dit-cpu-offload --text-encoder-cpu-offload ``` Notes on launch behavior: * **Default endpoint** is `127.0.0.1:30000` (`--host` / `--port` override). * **CPU offload flags are optional.** `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--image-encoder-cpu-offload` are available; defaults are auto-adjusted from GPU memory (GPUs under 30 GB get more aggressive offloading). * **Multi-GPU realtime.** Prefer explicit sequence parallelism (`--sp-degree` equal to the number of GPUs for a single session). Do not enable CFG parallel for the realtime profile: the default realtime request uses `guidance_scale=1.0`, while CFG parallel requires active cond/uncond branches. * **FSDP.** Use `--use-fsdp-inference` only when you specifically need weight sharding for memory. For the low-latency realtime profile, prefer keeping components resident and using SP first. * **Warmup.** Server warmup is **automatically skipped** for the realtime pipeline — a synthetic warmup request has no WebSocket session, so the server detects the registered realtime adapter and skips it. No explicit `--warmup-mode` setting is needed. Once up, the realtime WebSocket endpoint lives at `ws://127.0.0.1:30000/v1/realtime_video/generate` (use the Python client in §7 to connect — plain `curl` does not speak the `ws://` upgrade). ## 7. Realtime WebSocket API The realtime API is a single WebSocket at **`/v1/realtime_video/generate`**. All messages — client → server and server → client — are **msgpack** (`msgspec.msgpack.encode` / `decode`), not JSON. The lifecycle is: The client opens the WebSocket and sends exactly one **init** message (`type: "init"`), carrying the prompt, the required `first_frame`, output/sampling options, and optional camera conditions in `condition_inputs`. While generation runs, the client may push **event** messages (`type: "event"`) to steer the camera — either `kind: "camera_actions"` (frame-by-frame lists or state transitions) or `kind: "action"` (an action-DSL string). The server streams **frame batches** back. Each chunk arrives as one or more `frame_batch` messages (header fields + payload bytes); `is_final_frame_batch: true` marks the end of a chunk. The server also emits `chunk_stats` timing messages. ### INIT message `RealtimeVideoGenerationsRequest` (`type` is the literal `"init"`). Key fields: | Field | Type | Notes | | ------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `type` | `"init"` | Required literal | | `prompt` | str | Text prompt | | `first_frame` | bytes \| str | **Required by the SANA-WM adapter** (`on_init` raises if absent), though the generic request schema defines it as optional. Raw image bytes, a server-side path, or an `http(s)://` URL (downloaded & cached) | | `condition_inputs` | dict | Camera/conditioning inputs (see below) | | `num_frames` | int | Total frames to generate. **Omit it for an open-ended, continuous session** — the adapter leaves `num_frames` unset and flags an open-ended run (`condition_inputs["sana_wm_open_ended"] = True`), generating uniform chunks indefinitely (until `max_chunks` or the client disconnects). Provide an integer for a fixed-length clip | | `seed` | int | RNG seed (default `42`) | | `size` | str | `"WIDTHxHEIGHT"`; realtime requests default to `"832x480"` for latency. Pass `"1280x704"` for the native landscape resolution | | `max_chunks` | int | Optional cap on total chunks generated | | `num_inference_steps` | int | Default `4` for SANA-WM (realtime adapter) | | `guidance_scale` | float | Default `1.0` | | `realtime_output_format` | `"raw"` \| `"webp"` \| `"jpeg"` | Frame encoding for output (see below) | | `realtime_causal_sink_size` | int | Optional override | | `realtime_causal_kv_cache_num_frames` | int | Optional override | `condition_inputs` accepts (all optional; pass **only one** of `action` / `camera_actions`): | Key | Type | Meaning | | ----------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `camera_actions` | `list[list[str]]` or `{mode: "state", transitions: [...]}` | Frame-by-frame camera actions, or state-based transitions | | `action` | str | Action-DSL string, e.g. `"w-10,none-5,a-8"` (see §8) | | `intrinsics_path` | str | Server-side path to a camera-intrinsics **`.npy`** file (loaded via `np.load`; shapes `(4,)`, `(3,3)`, or `(F,3,3)`) | | `intrinsics` | list | Inline intrinsics with shape `(4,)`, `(3,3)`, `(F,4)`, or `(F,3,3)` | If you omit both `intrinsics_path` and `intrinsics`, SGLang uses a centered heuristic intrinsic matrix derived from the first-frame size. Pass explicit intrinsics when you need closer camera parity with a prepared trajectory. ```json INIT (msgpack dict) — open-ended (omit num_frames) theme={null} { "type": "init", "prompt": "beautiful landscape video", "first_frame": "", "size": "832x480", "seed": 42, "max_chunks": 10, "realtime_output_format": "raw", "num_inference_steps": 4, "guidance_scale": 1.0, "condition_inputs": { "camera_actions": [["w"], [], ["a", "s"]], "intrinsics_path": "/path/to/intrinsics.npy" } } ``` ### Live EVENT messages `RealtimeEvent` (`type: "event"`). Use `kind` + `payload` (optional `event_id` correlates the response back to this event). ```json EVENT - camera_actions (frame-by-frame list[list[str]]) theme={null} { "type": "event", "kind": "camera_actions", "event_id": 1, "payload": [["w"], ["w"], ["a"], []] } ``` ```json EVENT - camera_actions (state-based transitions) theme={null} { "type": "event", "kind": "camera_actions", "event_id": 2, "payload": { "mode": "state", "transitions": [ {"actions": ["w"], "client_ts_ms": 1000}, {"actions": ["a", "w"], "client_ts_ms": 1500} ] } } ``` ```json EVENT - action (DSL string) theme={null} { "type": "event", "kind": "action", "event_id": 3, "payload": "w-10,none-5,a-8,d-10" } ``` ### Server frame output The server streams **frame batches**. Every batch arrives as a **single** msgpack message with `type: "frame_batch"` — the header fields below plus an inline `payload` bytes field (the wire `type` is always `"frame_batch"`; there is no separate header-then-bytes message). Header fields: | Field | Meaning | | ---------------------------------------- | ------------------------------------------------------------------------------------------ | | `type` | `"frame_batch"` (always) | | `request_id` | Generation id | | `chunk_index` | Chunk index | | `content_type` | `application/x-raw-rgb`, `application/x-raw-rgb-delta-gzip`, `image/webp`, or `image/jpeg` | | `num_frames` | Frames in this batch | | `total_size` | Payload size in bytes (`len(payload)` — the compressed size for delta-gzip) | | `width`, `height`, `channels` | Frame geometry (`channels: 3`) | | `bytes_per_frame` | Bytes per uncompressed frame (`width*height*3`) | | `format` | `rgb24` for raw | | `encoding` | `raw`, `delta-gzip`, `webp`, or `jpeg` | | `delta_reference` | `previous-frame` (present for delta-gzip) | | `event_id` | Echoes the steering event id; **omitted** from the header for INIT-only chunks | | `frame_batch_index`, `num_frame_batches` | Sequence multiple batches within a chunk | | `is_final_frame_batch` | `true` ends the chunk | ```json Server output - frame_batch (msgpack dict) theme={null} { "type": "frame_batch", "request_id": "uuid-string", "chunk_index": 0, "content_type": "application/x-raw-rgb-delta-gzip", "num_frames": 3, "total_size": 1048576, "width": 1280, "height": 704, "channels": 3, "bytes_per_frame": 2703360, "format": "rgb24", "encoding": "delta-gzip", "delta_reference": "previous-frame", "event_id": 1, "frame_batch_index": 0, "num_frame_batches": 1, "is_final_frame_batch": true, "payload": "" } ``` **Encodings.** `application/x-raw-rgb` is uncompressed RGB24 (3 × uint8, `bytes_per_frame = width*height*3`). `application/x-raw-rgb-delta-gzip` is the zlib-compressed **per-frame XOR delta** against the preceding frame (each frame in the batch is XOR'd against the previous one; sent by default). `realtime_output_format: "raw"` forces uncompressed RGB; `"webp"` / `"jpeg"` send preview-encoded frames. delta-gzip must be restored **frame-by-frame**: decompress the payload, then for each frame XOR it against the already-restored previous frame (the first frame of a batch references the last frame of the previous batch). See `restore_delta_gzip_raw_rgb_payload` in `runtime/utils/realtime_video.py`. The `"raw"` format below avoids this. ### Minimal client example ```python Python theme={null} import msgspec import numpy as np import websockets # pip install websockets WS_URL = "ws://127.0.0.1:30000/v1/realtime_video/generate" async def run(): async with websockets.connect(WS_URL, max_size=None) as ws: # 1) INIT — omit num_frames for an open-ended session; "raw" = uncompressed RGB24 with open("first_frame.png", "rb") as f: first_frame = f.read() await ws.send(msgspec.msgpack.encode({ "type": "init", "prompt": "a camera moving forward and turning right", "first_frame": first_frame, "size": "832x480", "seed": 42, "max_chunks": 10, "realtime_output_format": "raw", "num_inference_steps": 4, "guidance_scale": 1.0, "condition_inputs": { "action": "w-100,wd-50,d-30", "intrinsics_path": "/path/to/intrinsics.npy", # optional; centered heuristic if omitted }, })) # 2) optional: steer mid-stream await ws.send(msgspec.msgpack.encode({ "type": "event", "kind": "camera_actions", "event_id": 1, "payload": [["w"], ["w"], ["a"], []], })) # 3) receive frame batches (raw RGB24) async for message in ws: msg = msgspec.msgpack.decode(message) if msg.get("type") != "frame_batch": continue # skip chunk_stats etc. n, h, w, c = msg["num_frames"], msg["height"], msg["width"], msg["channels"] frames = np.frombuffer(msg["payload"], dtype=np.uint8).reshape(n, h, w, c) # ... display/save frames ... if msg.get("is_final_frame_batch") and msg.get("chunk_index", 0) >= 9: break # asyncio.run(run()) ``` ## 8. Camera Action DSL Camera trajectories are described by a compact string of comma-separated `-` segments, e.g. `"w-100,wd-50,d-30,none-10"`. This is the format accepted by `condition_inputs.action` at init and by `kind: "action"` events. Parsing rules (`parse_action_string`): * Each segment is `-`; `` must be a positive integer. * `none` means no motion for that span: `none-10` = 10 static frames. * Keys are case-insensitive; combined keys apply simultaneously (`wd` = forward + right strafe). Allowed keys are exactly `wasdijkl`. | Key | Motion | | --------- | ----------------------- | | `w` / `s` | move forward / backward | | `a` / `d` | strafe left / right | | `i` / `k` | look (pitch) up / down | | `j` / `l` | look (yaw) left / right | Pose generation (`action_string_to_c2w`): * **Translation** (`w`/`s`/`a`/`d`) moves at `translation_speed` (default `0.04` world-units/frame). * **Rotation** (`i`/`k` pitch, `j`/`l` yaw) turns at `rotation_speed_deg` (default `1.2`°/frame); pitch is clamped to ±85°. * **Strafe-yaw coupling** (coefficient `0.4`): a `d` (right) strafe also nudges yaw right and `a` (left) nudges yaw left, so `wd` traces a curving arc rather than a pure sidestep. * Produces `(F+1, 4, 4)` camera-to-world matrices; the realtime stage pads the trajectory to the requested frame count. Example: `"w-100,wd-50,d-30,none-10"` = 100 frames forward → 50 frames forward + sweep right → 30 frames right strafe → 10 frames static. ## 9. Configuration Reference SANA-WM's defaults live in three places: **request-time** sampling params, the **pipeline config** (streaming/refiner knobs), and the **realtime adapter** (init-time overrides). ### Request-time — `SanaWMSamplingParams` (`configs/sample/sana_wm.py`) | Field | Default | Purpose | | --------------------- | ------- | --------------------------------------------------------------------- | | `height` | `704` | Output height | | `width` | `1280` | Output width | | `num_frames` | `49` | Total pixel frames (must satisfy `(num_frames - 1) % 8 == 0`) | | `fps` | `16` | Output frame rate (overrides the base default of 24) | | `num_inference_steps` | `20` | Stage-1 step count | | `guidance_scale` | `4.5` | Dense-path CFG scale | | `negative_prompt` | `""` | Negative prompt | | `camera_to_world` | `None` | In-memory `(T,4,4)` c2w extrinsics (mutually exclusive with `action`) | | `intrinsics` | `None` | In-memory `(T,3,3)` pinhole intrinsics | | `action` | `None` | Action-DSL string (see §8) | | `translation_speed` | `0.04` | World-units/frame for W/S/A/D | | `rotation_speed_deg` | `1.2` | Degrees/frame for I/K/J/L | | `pitch_limit_deg` | `85.0` | Pitch clamp | `generator_device` is inherited from the base `SamplingParams` (default `None` = use the pipeline/model default). On the `/v1/videos` HTTP API the camera fields are passed inside `diffusers_kwargs` (`action` / `intrinsics`, as in §4–5). ### Pipeline config — `SanaWMPipelineConfig` (`configs/pipeline_configs/sana_wm.py`) These are server-launch knobs (set via the `--streaming` / `--refiner-chunked` / `--num-frame-per-block` CLI flags or a pipeline-config override), **not** request fields: | Field | Default | Purpose | | ----------------------- | -------------------------- | -------------------------------------------------------- | | `streaming` | `False` | Chunk-causal `forward_long` (§5) vs dense one-shot (§4) | | `refiner_chunked` | `True` | Chunk-wise streaming refiner vs whole-clip dense refiner | | `num_frame_per_block` | `3` | Latent frames per Stage-1 / refiner chunk | | `num_cached_blocks` | `2` | Rolling KV-cache history window | | `denoising_step_list` | `(1000, 960, 889, 727, 0)` | 4-step streaming self-forcing timesteps (must end in 0) | | `streaming_cfg_scale` | `1.0` | CFG scale for the distilled streaming path (1.0 = off) | | `sink_size` | `1` | Sink (unrefined context) frames | | `refiner_block_size` | `3` | Refiner block size | | `refiner_kv_max_frames` | `11` | Refiner sliding KV window | ### Realtime adapter init overrides — `SanaWMRealtimeAdapter` At WebSocket `init` the realtime adapter fills SANA-WM defaults that differ from the request/sampling defaults above: | Field | Realtime default | Note | | --------------------- | ---------------- | --------------------------------------------------------------------- | | `size` | `832x480` | Realtime request default; pass `1280x704` for native landscape output | | `num_frames` | *(unset)* | Omitting → open-ended continuous session (§7) | | `num_inference_steps` | `4` | Distilled few-step | | `guidance_scale` | `1.0` | CFG off | | `fps` | `16` | Native rate | `guidance_scale` applies to the dense path (§4) only; the distilled streaming path uses `streaming_cfg_scale` (default `1.0`, i.e. no CFG) so a `guidance_scale` override never accidentally enables CFG on the streaming stage. `denoising_step_list = (1000, 960, 889, 727, 0)` is the official 4-step streaming schedule (it must end in 0). # Wan2.1 Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.1 ## 1. Model Introduction [Wan2.1](https://github.com/Wan-Video/Wan2.1) is a broad open video family covering text-to-video and image-to-video across 1.3B and 14B checkpoints. Its practical strengths are motion-rich generation, temporal consistency, and readable Chinese/English text, with 480p and 720p variants for different quality and memory targets. Choose the 1.3B T2V model for consumer-GPU experiments and the 14B models when quality matters more than footprint. Wan2.1 is a dense DiT family; for timestep-specialized MoE capacity or the unified 5B TI2V path, use Wan2.2 instead. ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom. **Interactive Command Generator**: Use the configuration selector below to automatically generate an appropriate deployment command for your model variant and options. ### 3.2 Configuration Tips Current supported optimization options are listed in the [SGLang diffusion support matrix](../../../docs/sglang-diffusion/attention_backends#platform-support-matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID. If not specified, the VAE will be loaded from the main model path. * `--num-gpus {NUM_GPUS}`: Number of GPUs to use. * `--tp-size {TP_SIZE}`: Tensor parallelism size (for the encoder/DiT; keep (\leq 1) if relying heavily on CPU offload). * `--sp-degree {SP_SIZE}`: Sequence parallelism degree. * `--ulysses-degree {ULYSSES_DEGREE}`: Degree of DeepSpeed-Ulysses-style SP in USP. * `--ring-degree {RING_DEGREE}`: Degree of ring attention-style SP in USP. * `--text-encoder-cpu-offload`, `--dit-cpu-offload`, `--vae-cpu-offload`: Use CPU offload to reduce peak GPU memory when needed. ## 4. Model Invocation ### 4.1 Basic Usage For more API usage and request examples, please refer to: [SGLang Diffusion OpenAI API](../../../docs/sglang-diffusion/api/openai_api) #### 4.1.1 Launch a server and then send requests ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 curl http://127.0.0.1:30000/v1/images/generations \ -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "Wan-AI/Wan2.1-T2V-14B-Diffusers", "prompt": "A cute baby sea otter", "n": 1, "size": "1024x1024", "response_format": "b64_json" }' ``` #### 4.1.2 Generate a video without launching a server ```bash Command theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --enable-cfg-parallel ) SAMPLING_ARGS=( --prompt "A curious raccoon" --save-output --output-path outputs --output-file-name "A curious raccoon.mp4" ) sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve significant inference speedups with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Advanced Usage** Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` #### 4.2.2 GPU Optimization * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if you run out of memory with FSDP. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Use as a workaround if you see "CUDA error: invalid argument". #### 4.2.3 Supported LoRA Registry SGLang supports applying Wan2.1 LoRA adapters on top of base models:
origin model supported LoRA
[Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) [NIVEDAN/wan2.1-lora](https://huggingface.co/NIVEDAN/wan2.1-lora)
[Wan-AI/Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) [valiantcat/Wan2.1-Fight-LoRA](https://huggingface.co/valiantcat/Wan2.1-Fight-LoRA)
**Example**: ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 \ --lora-path NIVEDAN/wan2.1-lora ``` ## 5. Reference Benchmark The following benchmark is a point-in-time reference for one model, hardware platform, SGLang image, and parameter set. It is not a complete hardware support matrix. Test Environment: * Hardware: AMD MI300X GPU (1x) * Model: Wan-AI/Wan2.1-T2V-14B-Diffusers * SGLang Docker Image Version: 0.5.9 ### 5.1 How to Run Benchmarks with SGLang You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 performance on your hardware. #### 5.1.1 Generate a single video **Server Command**: ```bash Command theme={null} sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1958.41 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 1958.4059 Latency Median (s): 1958.4059 Latency P99 (s): 1958.4059 -------------------------------------------------- Peak Memory Max (MB): 59662.00 Peak Memory Mean (MB): 59662.00 Peak Memory Median (MB): 59662.00 ============================================================ ``` **Server Command**: ```bash Command theme={null} #One A3 card has 2 npu chips. Benchmark was did with two A3 cards sglang serve \ --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \ --tp-size 2 \ --sp-degree 2 \ --num-gpus 4 \ --attention-backend laser_attn ``` **Benchmark Command**: ```bash Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1282.90 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 1282.90 Latency Median (s): 1282.90 Latency P90 (s): 1282.90 Latency P95 (s): 1282.90 Latency P99 (s): 1282.90 -------------------------------------------------- Peak Memory Max (MB): 31938.00 Peak Memory Mean (MB): 31938.00 Peak Memory Median (MB): 31938.00 ============================================================ ``` #### 5.1.2 Generate videos with Cache-DiT acceleration **Server Command**: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers ``` **Benchmark Command**: ```bash Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers Dataset: vbench -------------------------------------------------- Benchmark duration (s): 556.99 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 556.9885 Latency Median (s): 556.9885 Latency P99 (s): 556.9885 -------------------------------------------------- Peak Memory Max (MB): 69306.00 Peak Memory Mean (MB): 69306.00 Peak Memory Median (MB): 69306.00 ============================================================ ``` **Server Command**: ```bash Command theme={null} #One A3 card has 2 npu chips. Benchmark was did with two Atlas 3 cards SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \ --tp-size 2 \ --sp-degree 2 \ --num-gpus 4 \ --attention-backend laser_attn ``` **Benchmark Command**: ```bash Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.1-T2V-14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 413.88 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 413.88 Latency Median (s): 413.88 Latency P90 (s): 413.88 Latency P95 (s): 413.88 Latency P99 (s): 413.88 -------------------------------------------------- Peak Memory Max (MB): 32782.00 Peak Memory Mean (MB): 32782.00 Peak Memory Median (MB): 32782.00 ============================================================ ``` # Wan2.2 Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.2 ## 1. Model Introduction [Wan2.2](https://github.com/Wan-Video/Wan2.2) extends the Wan family with timestep-specialized Mixture-of-Experts models and a unified 5B TI2V checkpoint. It targets cinematic composition, lighting, and complex motion while supporting text-to-video, image-to-video, and hybrid text/image conditioning. Choose the A14B MoE checkpoints for maximum T2V or I2V capacity and the 5B TI2V model for a smaller unified 720p-at-24-fps path. MoE reduces active compute relative to total capacity but does not remove the memory cost of loading expert weights, so hardware selection still matters. ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus {NUM_GPUS}`: Number of GPUs to use * `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP ## 4. Model Invocation ### 4.1 Basic Usage For more API usage and request examples, please refer to: [SGLang Diffusion OpenAI API](/docs/sglang-diffusion/api/openai_api) #### 4.1.1 Launch a server and then send requests ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 curl http://127.0.0.1:3000/v1/images/generations \ -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers", "prompt": "A cute baby sea otter", "n": 1, "size": "1024x1024", "response_format": "b64_json" }' ``` #### 4.1.2 Generate a video without launching a server ```shell Command theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --enable-cfg-parallel ) SAMPLING_ARGS=( --prompt "A curious raccoon" --save-output --output-path outputs --output-file-name "A curious raccoon.mp4" ) sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}" ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` #### 4.2.2 GPU Optimization * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory with FSDP. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. Enable if run out of memory with FSDP. * `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. Enable if run out of memory with FSDP. * `--vae-cpu-offload`: Use CPU offload for VAE. Enable if run out of memory. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". #### 4.2.3 Supported LoRA Registry
origin model supported LoRA
[Wan-AI/Wan2.2-I2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers) [lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)
[Wan-AI/Wan2.2-T2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers) [Cseti/wan2.2-14B-Arcane\_Jinx-lora-v1](https://huggingface.co/Cseti/wan2.2-14B-Arcane_Jinx-lora-v1)
**Example**: ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 \ --lora-path Cseti/wan2.2-14B-Arcane_Jinx-lora-v1 ``` ## 5. Benchmark Test Environment: * Hardware: NVIDIA B200 GPU (1x) * Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers * sglang diffusion version: 0.5.6.post2 ### 5.1 Speedup Benchmark ### 5.1.1 Generate a video **Server Command**: ```shell Command theme={null} sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench Task: text-to-video -------------------------------------------------- Benchmark duration (s): 630.43 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 630.4277 Latency Median (s): 630.4277 Latency P99 (s): 630.4277 -------------------------------------------------- Peak Memory Max (MB): 62627.41 Peak Memory Mean (MB): 62627.41 Peak Memory Median (MB): 62627.41 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 card has 2 npu chips. Using four A3 cards in benchmarking sglang serve \ --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \ --tp-size 2 \ --sp-degree 4 \ --num-gpus 8 \ --attention-backend laser_attn ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 1 \ --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 214.50 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 214.50 Latency Median (s): 214.50 Latency P90 (s): 214.50 Latency P95 (s): 214.50 Latency P99 (s): 214.50 -------------------------------------------------- Peak Memory Max (MB): 46692.00 Peak Memory Mean (MB): 46692.00 Peak Memory Median (MB): 46692.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate videos with high concurrency **Server Command**: ```shell Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers Dataset: vbench Task: text-to-video -------------------------------------------------- Benchmark duration (s): 5163.21 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.00 Latency Mean (s): 2739.7695 Latency Median (s): 2742.0673 Latency P99 (s): 5121.6331 -------------------------------------------------- Peak Memory Max (MB): 72523.56 Peak Memory Mean (MB): 70253.34 Peak Memory Median (MB): 70824.46 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 card has 2 npu chips. Using four A3 cards in benchmarking SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang serve \ --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \ --tp-size 2 \ --sp-degree 4 \ --num-gpus 8 \ --attention-backend laser_attn ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench \ --task text-to-video \ --num-prompts 20 \ --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-video Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers/ Dataset: vbench -------------------------------------------------- Benchmark duration (s): 4384.65 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 Completed outputs: 20 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.00 Output throughput (outputs/s): 0.00 Latency Mean (s): 2304.17 Latency Median (s): 2297.69 Latency P90 (s): 3972.32 Latency P95 (s): 4178.99 Latency P99 (s): 4343.52 -------------------------------------------------- Peak Memory Max (MB): 46692.00 Peak Memory Mean (MB): 46691.90 Peak Memory Median (MB): 46692.00 ------------------------------------------------------------ ``` # Z-Image-Turbo Source: https://docs.sglang.io/cookbook/diffusion/Z-Image/Z-Image-Turbo ## 1. Model Introduction [Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled 6B single-stream DiT for fast text-to-image generation. It reaches its intended operating point in 8 function evaluations and is particularly strong at photorealistic scenes, prompt adherence, and English/Chinese text rendering. Choose it when latency and a relatively small deployment footprint matter more than the editability or maximum capacity of larger image models. It is a generation-only checkpoint; use Qwen-Image-Edit or FLUX.2 when the request includes source images or identity-preserving edits. ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. Please refer to the [official SGLang-diffusion installation guide](https://docs.sglang.io/docs/sglang-diffusion/installation) for installation instructions. ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. ### 3.2 Configuration Tips Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). * `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. * `--num-gpus`: Number of GPUs to use * `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) * `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs) * `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP * `--ring-degree`: The degree of ring attention-style SP in USP **AMD ROCm Notes**: Requires SGLang >= v0.5.8. ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). ### 4.1 Generate an Image ```python Example theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") response = client.images.generate( model="Tongyi-MAI/Z-Image-Turbo", prompt="A logo With Bold Large text: SGL Diffusion", n=1, response_format="b64_json", ) # Save the generated image image_bytes = base64.b64decode(response.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` ### 4.2 Advanced Usage #### 4.2.1 Cache-DiT Acceleration SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit). **Basic Usage** ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Tongyi-MAI/Z-Image-Turbo ``` **Advanced Usage** * DBCache Parameters: DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
Combined Configuration Example: ```bash Command theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang serve --model-path Tongyi-MAI/Z-Image-Turbo ``` #### 4.2.2 CPU Offload * `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory. * `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. * `--vae-cpu-offload`: Use CPU offload for VAE. * `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". ## 5. Benchmark Test Environment: * Hardware: AMD Instinct MI300X GPU (1x) * Model: Tongyi-MAI/Z-Image-Turbo * Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x * sglang diffusion version: 0.5.8 ### 5.1 Speedup Benchmark #### 5.1.1 Generate an image **Server Command**: ```shell Command theme={null} sglang serve --model-path Tongyi-MAI/Z-Image-Turbo \ --ulysses-degree=1 --ring-degree=1 --port 30000 ``` **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 1.84 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 -------------------------------------------------- Request throughput (req/s): 0.54 Latency Mean (s): 1.8435 Latency Median (s): 1.8435 Latency P99 (s): 1.8435 -------------------------------------------------- Peak Memory Max (MB): 30689.20 Peak Memory Mean (MB): 30689.20 Peak Memory Median (MB): 30689.20 ============================================================ ``` **Server Command**: ```shell Command theme={null} #One A3 card has 2 npu chips sglang serve --model-path Tongyi-MAI/Z-Image-Turbo --tp-size 2 --sp-degree 1 --num-gpus 2 ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 2.43 Request rate: inf Max request concurrency: 1 Successful requests: 1/1 Completed outputs: 1 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.41 Output throughput (outputs/s): 0.41 Latency Mean (s): 2.43 Latency Median (s): 2.43 Latency P90 (s): 2.43 Latency P95 (s): 2.43 Latency P99 (s): 2.43 -------------------------------------------------- Peak Memory Max (MB): 11052.00 Peak Memory Mean (MB): 11052.00 Peak Memory Median (MB): 11052.00 ------------------------------------------------------------ ``` #### 5.1.2 Generate images with high concurrency **Benchmark Command**: ```shell Command theme={null} python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: Tongyi-MAI/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 35.32 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 -------------------------------------------------- Request throughput (req/s): 0.57 Latency Mean (s): 18.5672 Latency Median (s): 18.5573 Latency P99 (s): 34.9880 -------------------------------------------------- Peak Memory Max (MB): 30689.26 Peak Memory Mean (MB): 30689.21 Peak Memory Median (MB): 30689.21 ============================================================ ``` **Benchmark Command**: ```shell Command theme={null} python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 ``` **Result**: ```text Output theme={null} ================= Serving Benchmark Result ================= Task: text-to-image Model: /models/Tongyi-MAI/Z-Image-Turbo/Z-Image-Turbo Dataset: vbench -------------------------------------------------- Benchmark duration (s): 49.08 Request rate: inf Max request concurrency: 20 Successful requests: 20/20 Completed outputs: 20 Outputs per prompt: 1 -------------------------------------------------- Request throughput (req/s): 0.41 Output throughput (outputs/s): 0.41 Latency Mean (s): 25.78 Latency Median (s): 25.77 Latency P90 (s): 44.42 Latency P95 (s): 46.75 Latency P99 (s): 48.61 -------------------------------------------------- Peak Memory Max (MB): 11054.00 Peak Memory Mean (MB): 11054.00 Peak Memory Median (MB): 11054.00 ------------------------------------------------------------ ``` # Overview Source: https://docs.sglang.io/cookbook/diffusion/intro Practical guides for deploying and using diffusion models with SGLang. Choose a recipe by output modality. The sidebar stays organized by model family, while this overview separates image, video, and realtime/world workloads. ## Image Models Image models generate one image request as a bounded denoising job, usually with bidirectional attention over the whole latent sequence. ## Video Models Video models denoise a bounded latent video sequence for each request. Use these recipes for offline text-to-video, image-to-video, and video generation serving. ## Realtime / World Models Realtime models keep a session alive and generate chunk by chunk with causal state, control signals, and cached video history. Use the sidebar group for LingBot World family variants. The overview links the newer LingBot World 2.0 recipe directly. # SGLang Cookbook Source: https://docs.sglang.io/cookbook/intro A community-maintained repository of practical guides and recipes for deploying and using SGLang in production environments. Our mission is simple: answer the question **"How do I use SGLang (and related models) on hardware Y for task Z?"** with clear, actionable solutions. ## Guides ## Benchmarks # SpecBundle Usage Source: https://docs.sglang.io/cookbook/specbundle/specbundle_usage specbundle logo ## About SpecBundle Speculative decoding, especially EAGLE3, offer strong theoretical guarantees alongside consistent empirical improvements in token acceptance rate and end-to-end inference speed. However, despite these advances, adoption of speculative decoding—especially EAGLE3—remains limited in the open-source ecosystem, due primarily to three key factors. 1. Lack of production-ready training infrastructure: Existing speculative decoding toolchains are largely research prototypes, offering limited system-level optimization and inadequate support for diverse architectures and large-scale models. 2. Scarcity of high-quality draft models: Effective speculative decoding depends on strong draft models, yet publicly available EAGLE3-compatible checkpoints are extremely limited, primarily originating from the original authors. 3. Insufficient training scale of existing drafts: Most available draft models are trained on small or curated datasets and fail to generalize to the large, diverse corpora used in modern LLM training, resulting in low token acceptance rates and diminished practical speedups. **SpecBundle** is a direct response to these limitations. Jointly driven by the open-source community and industry partners including **Ant Group**, **Meituan**, **Nex-AGI** and **EigenAI**, **SpecBundle** represents the **first open initiative** aimed at democratizing speculative decoding by providing high-performance, production-grade EAGLE3 draft model weights for mainstream open-source LLMs. This initiative also serves to verify the robustness of the [**SpecForge**](https://github.com/sgl-project/SpecForge) framework through multiple scales and architectures. ## Installation ```bash Command theme={null} git clone https://github.com/sgl-project/SpecForge.git ``` ## Usage ### Launch SGLang Server with SpecBundle models You can use the following command to launch the SGLang server with SpecBundle models. Please add `--tp`, `--ep` and `--mem-fraction-static` arguments when you encounter memory issues. ```bash Command theme={null} python3 -m sglang.launch_server \ --model \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ``` For example: ```bash Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` ### Use SpecBundle to compare the performance of Speculative Decoding draft models We provide a benchmark suite to evaluate the performance of SpecBundle draft models [here](https://github.com/sgl-project/SpecForge/tree/main/benchmarks). #### Example: 1. Launch a SGLang Server ```bash Command theme={null} SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp 4 ``` 2. Use the benchmark suite to evaluate the performance of SpecBundle draft models `bench_eagle3.py` can help you launch a SGLang server process and a Benchmarking process concurrently. In this way, you don't have to launch the SGLang server manually, this script will manually handle the SGLang launch under different speculative decoding configurations. Some important arguments are: * `--model-path`: the path to the target model. * `--speculative-draft-model-path`: the path to the draft model. * `--port`: the port to launch the SGLang server. * `--trust-remote-code`: trust the remote code. * `--mem-fraction-static`: the memory fraction for the static memory. * `--tp-size`: the tensor parallelism size. * `--attention-backend`: the attention backend. * `--config-list`: the list of speculative decoding configuration to test, the format is `,,,`. * `--benchmark-list`: the list of benchmarks to test, the format is `::`. ```bash Command theme={null} cd SpecForge/benchmarks python bench_eagle3.py \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \ --port 30000 \ --config-list 1,3,1,4 \ --benchmark-list mtbench:5 gsm8k:100 \ --skip-launch-server ``` **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate test command for your model and benchmark. It will generate a json file, content is listed below: ```json Config theme={null} { "mtbench": [ { "batch_size": 1, "steps": null, "topk": null, "num_draft_tokens": null, "metrics": [ { "latency": 12.232808108034078, "output_throughput": 319.71399906382845, "accept_length": 2.170366259711432, "accuracy": null, "num_questions": 5, "num_valid_predictions": 0, "categorical_performance": null } ], "num_samples": 5 } ], "gsm8k": [ { "batch_size": 1, "steps": null, "topk": null, "num_draft_tokens": null, "metrics": [ { "latency": 37.42077191895805, "output_throughput": 373.6160234823207, "accept_length": 2.643410852713178, "accuracy": 0.96, "num_questions": 100, "num_valid_predictions": 100, "categorical_performance": null } ], "num_samples": 100 } ] } ``` ## Performance Scores We evaluate the performance of SpecBundle draft models on various benchmarks, please visit the [Performance Dashboard](https://docs.sglang.io/SpecForge/SpecBundle/index.html) for more details. # Supported Models Source: https://docs.sglang.io/cookbook/specbundle/supported_models ## [Released Models](https://huggingface.co/collections/lmsys/specbundle) We list the models released by the SpecForge and several industrial partners below. These models are released as part of the SpecBundle models, which are trained on large-scale multi-domain datasets and deliver exceptional performance on various benchmarks. > We also include some of the models previously trained by the SpecForge team but not technically part of the SpecBundle release. > We mark models trained on ShareGPT+Ultrachat datasets with a **\*** mark and models trained on Perfect-Blend datasets but released before SpecBundle with **+** mark. ### Llama Series
Target Model EAGLE3 Draft Model
meta-llama/Llama-3.1-8B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.1-8B-Instruct-SpecForge)
meta-llama/Llama-3.3-70B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.3-70B-Instruct-SpecForge)
meta-llama/Llama-4-Scout-17B-16E-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-4-Scout-17B-16E-Instruct-SpecForge)
meta-llama/Llama-4-Maverick-17B-128E-Instruct [🤗 Hugging Face \*](https://huggingface.co/lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1)
### Qwen Series
Target Model EAGLE3 Draft Model
Qwen/Qwen3-30B-A3B-Instruct-2507 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex)
Qwen/Qwen3-235B-A22B-Instruct-2507 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan)
Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Next-80B-A3B-Instruct-FP8-perfect-blend-regenerated)
### Qwen Coder Series
Target Model EAGLE3 Draft Model
Qwen/Qwen3-Coder-30B-A3B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge)
Qwen/Qwen3-Coder-480B-A35B-Instruct [🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-480B-A35B-Instruct-SpecForge-EigenAI)
### Ling Series
Target Model EAGLE3 Draft Model
inclusionAI/Ling-flash-2.0 [🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Ling-Flash-2.0-eagle3)
### Kimi Series
Target Model EAGLE3 Draft Model
moonshotai/Kimi-K2-Instruct [🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Kimi-K2-Instruct-eagle3)
### GPT-OSS Series
Target Model EAGLE3 Draft Model
openai/gpt-oss-20b [🤗 Hugging Face +](https://huggingface.co/zhuyksir/EAGLE3-gpt-oss-20b-bf16)
openai/gpt-oss-120b [🤗 Hugging Face +](https://huggingface.co/lmsys/EAGLE3-gpt-oss-120b-bf16)
### Nex Series
Target Model EAGLE3 Draft Model
nex-agi/Qwen3-30B-A3B-Nex-N1 [🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-30B-A3B-Nex-N1)
nex-agi/Qwen3-32B-Nex-N1 [🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-32B-Nex-N1)
# Pi0.5 Source: https://docs.sglang.io/cookbook/vla/OpenPI/Pi0.5
dVLA OpenPI / LeRobot flow matching action robot edge
## 1. Model Introduction Pi0.5 is an OpenPI / LeRobot diffusion Vision-Language-Action (dVLA) policy. It consumes camera images, a language instruction, and robot state, then returns a continuous action chunk for robot control. SGLang serves Pi0.5 through the native `multimodal_gen` runtime. The implementation uses a SigLIP/PaliGemma prefix encoder and a Gemma action expert: the prefix is encoded once, then the action expert runs the flow-matching denoising loop. This is not a token decode workload, so the Pi0.5 path does not use the LLM sampler, logits processor, token streaming, paged decode KV cache, or a separate SRT serving engine. The SigLIP vision tower, PaliGemma language stack, and action expert are all SGLang-native modules. Transformers is used for checkpoint configuration and tokenization, not for the runtime neural network. The prefix encoder covers both stages of observation encoding: SigLIP turns resized camera pixels into continuous patch embeddings, then the PaliGemma transformer jointly encodes those patches with tokenized task/state inputs and produces per-layer prefix K/V. At flow timestep `t`, the action expert projects the noisy continuous action chunk `x_t` into action embeddings. Its queries attend to both the fixed prefix K/V and the current action K/V, while the timestep follows a separate sinusoidal-MLP path and conditions every action-expert layer through AdaRMSNorm gates. Supported public checkpoints: | Checkpoint | Cameras | State Dim | Output Action Dim | Action Horizon | Denoise Steps | | -------------------------- | ----------------------------------------------------- | --------: | ----------------: | -------------: | ------------: | | `lerobot/pi05_base` | `base_0_rgb`, `left_wrist_0_rgb`, `right_wrist_0_rgb` | 32 | 32 | 50 | 10 | | `lerobot/pi05_libero_base` | `image`, `image2`, one empty camera | 8 | 7 | 50 | 10 | References: * [OpenPI](https://github.com/Physical-Intelligence/openpi) * [lerobot/pi05\_base](https://huggingface.co/lerobot/pi05_base) * [LeRobot Pi0.5 docs](https://huggingface.co/docs/lerobot/en/pi05) ## 2. Installation Install SGLang with the diffusion extra. Pi0.5 lives in `multimodal_gen`, and the extra includes the runtime dependencies used by the policy server. ```bash Command theme={null} git clone https://github.com/sgl-project/sglang.git cd sglang pip install -e "python[diffusion]" ``` For general environment setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). ## 3. Model Deployment Serve the base Pi0.5 policy: ```bash Command theme={null} sglang serve lerobot/pi05_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` Serve the LIBERO checkpoint: ```bash Command theme={null} sglang serve lerobot/pi05_libero_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` These registered LeRobot checkpoints dispatch to the native `multimodal_gen` Pi0.5 pipeline automatically. The `--pipeline` / `--pipeline-class-name` flag is only an advanced override for local or private checkpoints that cannot be resolved from the model registry. Pi0.5 serving does not start a second SRT LLM serving engine. ### 3.1 Action Request Schema | Field | Type | Description | | -------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | string, optional | Served model name. When omitted, the server uses the currently loaded policy. | | `input.task` or `input.prompt` | string | Language instruction for the policy. | | `input.observation.images` | object | Map from camera name to RGB image. Images can be nested here or sent as OpenPI observation keys such as `observation.images.base_0_rgb` through the websocket adapter. | | `input.observation.state` | array or tensor object | Robot state vector. Use the same normalization convention as the OpenPI / LeRobot checkpoint. | | `input.observation.noise` | array or tensor object, optional | Initial action noise with shape `[action_horizon, action_dim]`. Use this for deterministic debugging. | | `parameters.num_inference_steps` | integer, optional | Flow-matching denoise steps. Defaults to `10`. | | `parameters.action_horizon` | integer, optional | Output action horizon. Defaults to the checkpoint config. | | `parameters.action_dim` | integer, optional | Internal padded action dimension. Defaults to the checkpoint config. | | `runtime.return_timing` | boolean, optional | Return stage timing fields. Defaults to `true`. | | `runtime.prefix_cache` | boolean or `"auto"`, optional | Enable exact full-prefix lookup for this request when the server has `enable_global_prefix_cache=true`. Defaults to `"auto"`. | | `runtime.cuda_graph` | boolean or `"auto"`, optional | Enable the available prefix and action-denoise CUDA graph paths for this request. Defaults to `"auto"`. | | `runtime.output_format` | `"list"` or `"numpy"`, optional | Use `"list"` for JSON compatibility. Use `"numpy"` with msgpack or Python clients to avoid Python-list materialization. Defaults to `"list"`. | | `runtime.response_format` | `"envelope"` or `"raw"`, optional | HTTP-only response shape. `"envelope"` returns the generic action envelope. `"raw"` returns the policy payload directly. Defaults to `"envelope"`. | ## 4. API Usage ### 4.1 Generic Action HTTP API Use `/v1/actions/generations` for direct policy calls and debugging. Use `/v1/actions/metadata` to discover the camera keys, state size, action shape, defaults, and websocket capabilities of the currently served policy. ```python Example theme={null} import numpy as np import requests image = np.zeros((224, 224, 3), dtype=np.uint8) payload = { "model": "lerobot/pi05_base", "input": { "task": "pick up the block", "observation": { "images": { "base_0_rgb": image.tolist(), "left_wrist_0_rgb": image.tolist(), "right_wrist_0_rgb": image.tolist(), }, "state": np.zeros(32, dtype=np.float32).tolist(), }, }, "runtime": { "return_timing": True, "prefix_cache": "auto", "cuda_graph": "auto", }, } response = requests.post( "http://127.0.0.1:30000/v1/actions/generations", json=payload, timeout=60, ) response.raise_for_status() data = response.json() actions = data["data"][0]["action"]["values"] print(len(actions), len(actions[0])) print(data.get("timings")) ``` The same `/v1/actions/generations` endpoint also accepts `Content-Type: application/msgpack` and can return msgpack when `Accept: application/msgpack` is set. For msgpack clients, send numpy arrays directly using the `pack_numpy_payload` helper from the websocket example below. Msgpack requests default to numpy action output on the server side; set `runtime.output_format` to `"list"` only when a client explicitly needs nested Python lists. For the lowest-overhead generic HTTP path, use msgpack with the raw policy response: ```python Example theme={null} payload["runtime"] = { "return_timing": True, "prefix_cache": False, "cuda_graph": "auto", "response_format": "raw", } response = requests.post( "http://127.0.0.1:30000/v1/actions/generations", data=packb(payload), headers={ "Content-Type": "application/msgpack", "Accept": "application/msgpack", }, timeout=60, ) result = unpackb(response.content) actions = result["actions"] ``` For `lerobot/pi05_libero_base`, use the LIBERO camera names and an 8-dimensional state vector: ```python Example theme={null} payload = { "input": { "task": "pick up the object", "observation": { "images": { "image": image.tolist(), "image2": image.tolist(), }, "state": np.zeros(8, dtype=np.float32).tolist(), }, }, } ``` ### 4.2 Generic Realtime WebSocket `/v1/actions/realtime` is the generic msgpack websocket path. It sends `action.metadata` on connect and returns the same `action.generation` envelope as the HTTP API for each request. ### 4.3 OpenPI-Compatible WebSocket Robot clients can use the OpenPI-compatible msgpack websocket endpoint at `/openpi/policy`. The server sends metadata immediately after connection, then each client message should contain one observation. ```python Example theme={null} import asyncio import msgpack import numpy as np import websockets def pack_array(obj): if isinstance(obj, np.ndarray): return { b"__ndarray__": True, b"data": obj.tobytes(), b"dtype": obj.dtype.str, b"shape": obj.shape, } if isinstance(obj, np.generic): return { b"__npgeneric__": True, b"data": obj.item(), b"dtype": obj.dtype.str, } return obj def unpack_array(obj): ndarray_marker = obj.get("__ndarray__") or obj.get(b"__ndarray__") npgeneric_marker = obj.get("__npgeneric__") or obj.get(b"__npgeneric__") data = obj.get("data", obj.get(b"data")) dtype = obj.get("dtype", obj.get(b"dtype")) shape = obj.get("shape", obj.get(b"shape")) if ndarray_marker: return np.ndarray( buffer=data, dtype=np.dtype(dtype), shape=shape, ) if npgeneric_marker: return np.dtype(dtype).type(data) return obj def packb(payload): return msgpack.packb(payload, default=pack_array, use_bin_type=True) def unpackb(payload): return msgpack.unpackb(payload, object_hook=unpack_array, raw=False) async def main(): image = np.zeros((224, 224, 3), dtype=np.uint8) observation = { "task": "pick up the block", "observation.images.base_0_rgb": image, "observation.images.left_wrist_0_rgb": image, "observation.images.right_wrist_0_rgb": image, "observation.state": np.zeros(32, dtype=np.float32), } async with websockets.connect( "ws://127.0.0.1:30000/openpi/policy", max_size=None, ) as websocket: metadata = unpackb(await websocket.recv()) print(metadata) await websocket.send(packb(observation)) result = unpackb(await websocket.recv()) actions = result["actions"] print(len(actions), len(actions[0])) print(result.get("server_timing")) asyncio.run(main()) ``` ## 5. Configuration Tips * Request-local `PrefixContext` is always reused across all denoise steps in one request. The prefix K/V is not cloned per step. * The optional global prefix cache is a bounded exact-match LRU. It is disabled by default because changing robot frames rarely hit it and enabling it prevents unrelated misses from entering grouped prefix execution. Set `enable_global_prefix_cache=true` for repeated observations, retries, or multiple policy calls over the same camera/state sample; `runtime.prefix_cache` can then disable lookup per request. * Partial-prefix reuse is not supported because Pi0.5 combines image and tokenized task/state inputs under full attention. Changing any input can change every deeper-layer prefix K/V tensor. The exact key hashes resized and normalized pixels before SigLIP, plus effective token IDs, token masks, camera masks, model revision, dtype, and parallel layout. Tensor content hashing reuses SRT's CPU/CUDA implementation; hashing the pre-SigLIP input lets an exact hit skip both the vision encoder and prefix transformer. * CUDA graph capture targets single-request prefix encoding and one action-denoise step. Prefix capture uses one bounded input-shape bucket by default; grouped prefixes, prefix TP, CPU offload, and global prefix-cache misses stay eager. The denoise graph is replayed across the flow-matching loop and uses batch size, prefix length, action horizon, action dim, dtype, and parallel layout in its shape signature. With action SP enabled, the denoise bucket uses the local action shard length and rank-specific position offset. * Cache-DiT is not used in the default Pi0.5 path. The current robot policy target is numerically lossless inference, while Cache-DiT-style reuse is an image/video DiT approximation that needs separate policy-quality validation before it can be recommended for action control. * Do not use CFG parallelism to split the 10 Euler steps. Use it only for independent branches such as multiple candidate actions or future conditional/unconditional branches. * Prefix TP uses native SGLang parallel linear layers for the PaliGemma language prefix model when model parallel TP is initialized and the VLA split broadcast group is not active. The action expert does not share that TP layout. The v1 split prefix/action path instead uses the SP group: prefix root computes/broadcasts `PrefixContext`, while action ranks run the SP action path. * The split path uses the SP group as the action group: prefix root computes/broadcasts `PrefixContext`, all action ranks broadcast the initial action noise once, shard the action horizon, and run the action expert through Ulysses attention when the prefix is full-attention, ring degree is one, heads are divisible by SP size, and the horizon is evenly shardable. Otherwise it falls back to action-root execution. * `lerobot/pi05_libero_base` returns 7 action dimensions even though the internal padded action tensor uses 32 dimensions. ## 6. VRAM Tuning For current public Pi0.5 checkpoints, a stable 16GB discrete GPU target is a reasonable v1 deployment bar for robot workstations. In practice, leave headroom for the driver, camera middleware, robot process, and allocator fragmentation. Jetson/Orin unified-memory devices need extra caution because system RAM and GPU memory share the same pool. OpenPI inference is mixed precision, not full fp32: most weights and compute run in `bf16`, selected stability-sensitive weights stay in `fp32`, and returned actions are `float32`. SGLang mirrors that policy by default. The validated `pi05_aloha` OpenPI PyTorch checkpoint keeps `119,720,608` parameters in fp32; SGLang reports the same fp32 stability set, plus `3,233,713,264` bf16 runtime parameters after skipping unused LM heads for continuous action inference. The fp32 set includes SigLIP patch/position embeddings, Gemma layer norms/final norms, and the action/time projection heads. Keep `materialize_dtype` at `bf16` unless you are debugging numerical parity. Current H100 pressure validation shows that the bf16 model path fits a 16GB-free budget without layerwise offload. The Pi0.5 path batches all camera frames for a grouped request into one SigLIP forward before splitting the embeddings back by camera, which is important for multi-camera robot workloads. The latency numbers below use the Python grouped API with global prefix cache disabled and CUDA graph enabled on unconstrained H100; rerun HTTP/OpenPI websocket on your target server before using the policy in a closed-loop robot. | Mode | Command Shape | Steady VRAM Snapshot | Notes | | ---------------------------- | -------------------------------------------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Single GPU bf16 | `--num-gpus 1`, no offload | fit with `16383 MiB` free before load in H100 pressure test | Recommended first path for 16GB-class discrete GPUs. | | ALOHA bf16 grouped | Python API, batch size 1 / 2 / 4 / 8 | unconstrained H100 | `52.4 ms` single; `65.1 ms / 2`; `91.9 ms / 4`; `147.4 ms / 8`. | | ALOHA 16GB-free pressure | Python API, no offload | same 16GB-free pressure | Fit was validated after the bf16 correction; rerun latency on the target because older pressure latency was collected before the final OpenPI precision fix. | | Global prefix cache disabled | `enable_global_prefix_cache=false` or request `runtime.prefix_cache=false` | avoids cache growth across changing frames | Recommended for tight edge budgets unless repeated exact frames are common. | | Offload fallback | config in 6.2 | target-dependent | Use only if the default bf16 path OOMs on the target device. | Use these knobs first because they do not change action numerics for a fixed input/noise: ### 6.1 16GB Edge Config Use this single-GPU config first for 16GB-class robot workstations. It keeps parameters resident on GPU, keeps CUDA graph enabled, and disables global prefix cache growth. ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": true, "prefix_cuda_graph_max_entries": 1, "enable_action_cuda_graph": true } ``` Start a single-GPU server with the override: ```bash Command theme={null} sglang serve lerobot/pi05_base \ --model-type diffusion \ --pipeline-config-path pi05_edge_16gb.json \ --num-gpus 1 \ --warmup-mode off \ --host 127.0.0.1 \ --port 30000 ``` Treat the H100 pressure result as a memory-budget validation, not a Jetson latency guarantee. Jetson/Orin devices use shared system memory and have much lower effective memory bandwidth than H100, so measure closed-loop control latency on the target device before deciding the action chunk cadence. ### 6.2 Offload Fallback Use offload only when the default bf16 config still does not fit on the target. These modes are numerically lossless for fixed inputs/noise, but they move weights between CPU and GPU and can hurt latency substantially. For a moderate fallback, offload cache growth and selected stage-resident modules first: ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": false, "enable_action_cuda_graph": false, "offload_prefix_image_encoder_after_embed": true, "offload_prefix_token_embedding": true, "offload_prefix_language_layer_count_after_prefix": 2, "offload_action_expert_after_denoise": true, "empty_cache_after_prefix": true } ``` If that still does not fit, full prefix layerwise CPU offload keeps every PaliGemma language layer on CPU and moves one layer at a time to GPU during prefix compute: ```json File theme={null} { "materialize_dtype": "bf16", "enable_global_prefix_cache": false, "prefix_cache_max_entries": 0, "enable_prefix_cuda_graph": false, "enable_action_cuda_graph": false, "offload_prefix_image_encoder": true, "offload_prefix_token_embedding": true, "offload_prefix_language_layers": true, "offload_prefix_language_layers_empty_cache": true, "empty_cache_after_prefix": true } ``` Offload validation should be repeated on the target hardware after any dtype or loader change. Earlier fp32-runtime offload numbers are not comparable to the current bf16 path. ### 6.3 Per-Request Controls For HTTP calls, disable cache or both CUDA graph paths without restarting the server: ```python Example theme={null} payload = { "input": { "task": "pick up the block", "observation": { "images": images, "state": state, }, }, "runtime": { "prefix_cache": False, "cuda_graph": False, }, } ``` For OpenPI websocket clients, include equivalent `enable_prefix_cache` and `enable_cuda_graph` fields in each raw msgpack observation if you need per-request compatibility controls. ### 6.4 Deployment Choices * Keep batch size to one control stream unless grouped robot streams are explicitly validated. More concurrent observations increase activation and PrefixContext residency. * Keep `materialize_dtype` at the default `bf16`. `fp32` is useful only for debugging numerical issues and will increase memory and latency. * Use split prefix/action only when you have multiple GPUs and have validated the robot control latency on that topology. On two GPUs with `--sp-degree 2 --ulysses-degree 2`, the action horizon can be sequence-sharded while the prefix root still computes and broadcasts `PrefixContext`. A single 16GB-class GPU should try the default bf16 config first. * Reducing `num_inference_steps` lowers latency but is not a pure memory fix and can change policy behavior. Validate closed-loop task success before using fewer than the checkpoint default. * CPU offload is a compatibility fallback, not the preferred 16GB path. Quantization and deeper prefix/vision sharding are the next steps for sub-16GB devices. ### 6.5 Loader And Run:ai Model Streamer Run:ai Model Streamer can improve cold-start and checkpoint loading by reading safetensors concurrently and streaming tensors toward GPU memory. It is useful for local SSD, object storage, and cloud deployments where startup time is dominated by model file IO. The `python[diffusion]` extra includes `runai_model_streamer`. If the package is installed, SGLang enables it by default through `SGLANG_USE_RUNAI_MODEL_STREAMER=true`. Set it to `false` to force the plain safetensors loader: ```bash Command theme={null} SGLANG_USE_RUNAI_MODEL_STREAMER=false \ sglang serve lerobot/pi05_base \ --model-type diffusion \ --host 127.0.0.1 \ --port 30000 ``` Pi0.5 uses the direct SSD-to-GPU Run:ai path when all load targets for the current process are GPU-resident. This check is rank-local: distributed action ranks can stream their action-expert subset directly to GPU, while any rank with CPU/offloaded target tensors stays on the CPU safetensors fallback. The single-GPU validation streamed `13.5 GiB` of safetensors directly to `cuda:0` in about `1.5 s`, then returned `[50, 32]` actions over both HTTP and OpenPI websocket. For mixed CPU offload and low-VRAM 16GB-class modes, Pi0.5 still uses the header-filtered safe loader for ranks whose target tensors are not fully CUDA-resident. Direct GPU streaming can increase the exact VRAM pressure the low-memory path is trying to avoid. When debugging distributed startup, compare with `SGLANG_USE_RUNAI_MODEL_STREAMER=false` to separate streamer behavior from model execution behavior. Run:ai Model Streamer does not reduce steady inference VRAM after parameters, caches, activations, CUDA contexts, and graph buffers are resident. For Pi0.5 low-VRAM work, prioritize component placement, prefix cache size, CUDA graph residency, and CPU/offload first. Model Streamer is a cold-start optimization after the steady-memory budget is correct. ## 7. OpenPI Comparison Benchmark Use `bench_pi05_openpi.py` when you need a side-by-side latency and action-difference report against the OpenPI policy implementation. Start the SGLang Pi0.5 server first for HTTP or websocket modes, then run the benchmark from the SGLang repository root: ```bash Command theme={null} python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \ --profile aloha \ --sglang-url http://127.0.0.1:30000 \ --openpi-checkpoint gs://openpi-assets/checkpoints/pi05_base \ --num-inference-steps 10 \ --batch-size 4 \ --repeats 20 \ --warmup 3 \ --deterministic-noise \ --output pi05_openpi_compare.json ``` The benchmark reports: * SGLang HTTP latency for `/v1/actions/generations`, including stage timings when the server returns them. * SGLang msgpack HTTP latency when `--sglang-api http_msgpack` is set. This keeps the generic HTTP endpoint but avoids JSON image-array overhead. Use `--sglang-http-response-format raw` to benchmark the compact raw policy response. * SGLang OpenPI-compatible websocket latency when `--sglang-api openpi_ws` is set. This uses persistent msgpack websocket connections and is closer to the robot client path than JSON-over-HTTP. * SGLang Python in-process latency when `--sglang-api python` is set. This loads the native Pi0.5 pipeline in the benchmark process and avoids HTTP, websocket, scheduler, and serialization overhead. Use `--sglang-python-batch-mode grouped` to exercise the conservative native grouped-batch path. The Python path also reports actual SGLang module parameter dtype counts and example parameter names. * OpenPI single-request latency through `Policy.infer`. * Batch latency for grouped robot streams. SGLang uses concurrent HTTP requests in HTTP mode and persistent multi-connection msgpack calls in websocket mode. The Python backend can use true grouped model execution for fresh-prefix requests. OpenPI defaults to its internal direct model batch path because the public `Policy.infer` API is single-observation. * Action difference in normalized model space from identical OpenPI-transformed model inputs and noise. The check requires `--deterministic-noise` and fails when either `--action-max-abs-diff` or `--action-mean-abs-diff` is exceeded. This mode isolates model parity from robot-specific normalization and action postprocessing. The LIBERO policy returns only 10 actions after policy postprocessing, but its flow-matching model still generates a 50-step chunk; the benchmark compares that model output with SGLang before OpenPI unnormalization and horizon slicing. For one-sided 16GB-class checks, run each backend separately under the same VRAM pressure. The SGLang Python path accepts the same pipeline config override as serving: ```bash Command theme={null} python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \ --profile aloha \ --sglang-api python \ --sglang-python-batch-mode grouped \ --skip-openpi \ --sglang-pipeline-config-path pi05_edge_16gb.json \ --num-inference-steps 10 \ --batch-size 4 \ --repeats 5 \ --warmup 2 \ --deterministic-noise \ --disable-prefix-cache \ --disable-cuda-graph ``` Use `--skip-sglang --openpi-pytorch-compile-mode none` to measure an OpenPI eager baseline in the same environment. For a PyTorch-native OpenPI baseline, point `--openpi-checkpoint` at a checkpoint directory containing `model.safetensors` and the OpenPI `assets/` norm-stat tree. The `keep` mode preserves OpenPI's checkpoint default compile setting. The grouped Python path currently requires compatible fresh-prefix requests without split prefix/action workers or effective prefix-cache hits; other cases fall back to per-request execution. ## 8. Validation Notes The following checks were run on H100 GPUs with the native SGLang Pi0.5 path: | Check | Result | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `lerobot/pi05_base` direct end-to-end | Prefix length `968`, output shape `[1, 50, 32]`, peak allocated memory `12.817 GiB`. | | Official OpenPI parity | Against OpenPI PyTorch revision `15a9616`, with the same LeRobot checkpoint revision, observation, and noise: first-step velocity max/mean absolute difference `0.02677` / `0.00344`; production 10-step normalized action `0.00813` / `0.00092`. | | Action denoise CUDA graph | Eager 10-step denoise `125.4 ms`; steady graph replay `50.8 ms`; max output difference `0`. | | Prefix CUDA graph | On H200 with action graph already enabled, ALOHA batch=1 p50 improved from `48.04 ms` to `42.98 ms` (`1.118x`). Two observations at 5 and 10 steps were bit-exact with graph disabled. One prefix shape bucket added about `48.5 MiB`; batch=4 showed no benefit and stays eager. | | Exact full-prefix cache | First prefix pass about `203 ms`; exact cache hit prefix stage about `0.2 ms`. | | `lerobot/pi05_libero_base` direct end-to-end | Image keys `image`, `image2`, `empty_camera_0`; state dim `8`; output action dim `7`; output tensor shape `[1, 50, 32]`. | | Python grouped execution | ALOHA batch=4 grouped path measured `91.9 ms / 4` on current mixed precision: prefix `18.4 ms`, action denoise `61.2 ms`, preprocess about `2.4 ms` per request. Sequential Python loop batch=4 measured `211.9 ms / 4`. | | `sglang serve` HTTP | `/v1/actions/generations` returned action shape `[50, 32]`. JSON-over-HTTP remains compatible but image-array serialization dominates. Msgpack HTTP with prefix cache disabled measured `57.2 ms` single and `221.9 ms / 4` for the envelope response, and `56.4 ms` single and `219.7 ms / 4` for `runtime.response_format="raw"`. | | OpenPI websocket | `/openpi/policy` returned action shape `[50, 32]`; with persistent msgpack connections and prefix cache disabled, ALOHA measured about `77.0 ms` single and `162.0 ms / 4`. | | 2-GPU prefix/action split | Earlier split validation returned action shape `[50, 32]` and matched single-GPU HTTP with max absolute difference `0`. After the true action-SP change, rerun this check with `--num-gpus 2 --sp-degree 2 --ulysses-degree 2` and verify both action ranks enter denoise kernels. | | OpenPI/SGLang precision | Official OpenPI JAX inference restores the public GCS checkpoint as bf16 with selected fp32 stability compute and returns float32 actions. The converted OpenPI PyTorch `pi05_aloha` checkpoint keeps `119,720,608` fp32 stability params; SGLang reports the same fp32 set and `3,233,713,264` bf16 runtime params after skipping unused LM heads. | | Native attention dtype | Checkpoint source tensors may be fp32, but SGLang finalizes PiGemma and SigLIP compute dtype before native attention backend selection; backend logs showed `Using fa attention backend` for the PiGemma path in the prior run. | | 16GB-free Python pressure | With an H100 artificially constrained to `16381 MiB` free before model load, single-GPU bf16 no-offload Python grouped path completed without OOM. Re-run latency after precision or loader changes before using pressure numbers for deployment sizing. | | Low-VRAM switches | Disabling prefix cache prevents cache growth across changing robot frames. Prefix graph residency is bounded by `prefix_cuda_graph_max_entries` (default `1`); set `enable_prefix_cuda_graph=false` or the limit to `0` to save about `48.5 MiB` for the validated ALOHA bucket. Action graph can stay enabled when the action expert remains resident; disable both graph paths for offload fallback modes. | | Offload fallback | CPU/offload modes are retained as numerically lossless compatibility fallbacks, but earlier fp32-runtime offload latency numbers are stale after the bf16 dtype correction and should be revalidated before deployment decisions. | | Run:ai direct loader | Single-GPU serve streamed `13.5 GiB` safetensors to `cuda:0` in about `1.5 s` and returned `[50, 32]` actions. Distributed direct streaming is now rank-local and should be revalidated on the target split topology; offload ranks with CPU targets still use the safe loader. | | OpenPI comparison status | Official OpenPI GCS `pi05_base` is a JAX checkpoint; converted PyTorch eager was validated without `torch.compile`. On 80GB H100, ALOHA OpenPI PyTorch eager was about `125-130 ms` single and about `164 ms / 4` in the direct-model batch path. Current SGLang Python grouped measured `52.4 ms` single and `91.9 ms / 4`; JAX OpenPI was `53.0 ms` single and `59.5 ms / 2` in a short check. | Run a full HTTP or websocket smoke test in your robot deployment environment before using the policy in a closed-loop controller. # Overview Source: https://docs.sglang.io/cookbook/vla/intro Practical guides for deploying and using Vision-Language-Action policies with SGLang. Vision-Language-Action policies map camera observations, language instructions, and robot state into continuous action chunks. They share some runtime machinery with diffusion pipelines, but the user-facing workload is robot action inference rather than image or video generation. This section keeps VLA policies separate from the diffusion model cookbook so robot deployments can document their own input schemas, policy endpoints, cache behavior, and control-loop performance targets. ## OpenPI # Adaptive Speculative Decoding Source: https://docs.sglang.io/docs/advanced_features/adaptive_speculative_decoding Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime. It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal. ## Current support * Only `--speculative-algorithm EAGLE` or `EAGLE3` * Only `--speculative-eagle-topk 1` * If either condition is not met, SGLang falls back to static speculative settings ## Why adaptive steps help `speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload. * If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early. * If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted. * At **high batch sizes**, the cost of each wasted draft step is multiplied across all sequences in the batch, so the optimal step count is often lower than at low batch sizes. Real traffic often moves between high-acceptance and low-acceptance phases, and batch sizes vary continuously. Adaptive mode follows both signals at runtime instead of hard-coding a single global `num_steps`. ## Design overview The adaptive mechanism has three pieces: * `AdaptiveSpeculativeParams`: the EMA-based policy * `SpecRuntimeState`: the per-tier runtime state bundle * `AdaptiveController`: the coordinator that queries the policy for the current batch size and activates the matching runtime state ### Per-batch-size independent tracking The controller maintains **independent EMA trackers for each batch size range**, so observations at small BS don't pollute the large BS signal. Each BS range can have its own candidate steps, hysteresis thresholds, and ceiling coefficient. BS ranges are defined as lower bounds in the config file (e.g., keys `"1"` and `"8"` mean BS 1–7 uses one slot, BS 8+ uses another). `SpecRuntimeState` objects are shared across BS ranges with the same step count — each state owns CUDA graphs captured for the reachable padded batch sizes of that step. ```mermaid theme={null} --- title: "SpecRuntimeState — speculative_num_steps / speculative_num_draft_tokens" --- graph LR subgraph SR[" "] direction LR subgraph D["Draft stage"] direction TB d1[attn_backend] d2[cuda_graph] end subgraph V["Verify stage"] direction TB v1[attn_backend] v2[cuda_graph] end subgraph E["Extend stage"] direction TB e1[attn_backend] e2[cuda_graph] end end ``` This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture. ## Runtime flow The adaptive update happens in two places: 1. **Pre-draft**: query the optimal step for the current batch size and activate if different 2. **Post-verify**: update the matching BS slot's EMA with observed accept lengths ```mermaid theme={null} --- title: "EAGLEWorker.forward_batch_generation() — decode path" --- flowchart TD Z["⓪ activate_step_by_batch(batch_size)
query optimal step for current BS range, activate if different"] A["① draft(batch)
draft model multi-step generation with current tier"] B["② verify(batch, spec_info)
target model tree verification → produces num_correct_drafts_per_req"] C["③ forward_draft_extend_after_decode(batch)
draft model KV-cache catch-up"] D["④ adaptive_controller.on_verify_complete(num_correct_drafts_per_req, batch_size)
update EMA for matching BS slot, apply warmup / interval / hysteresis gates
if tier changed, select a pre-built state from pool"] E["worker.apply_runtime_state(state)"] Z --> A --> B --> C --> D --> E ``` > Tier switch happens after the current round completes. Backends and CUDA graphs are never swapped mid-round. ## How the policy decides After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the candidate tiers for the matching BS slot. The decision logic is intentionally conservative: * `warmup_batches` skips the first few batches * `update_interval` avoids switching every batch * `down_hysteresis` and `up_hysteresis` reduce oscillation * `ceiling_coeff` — an optional EMA ceiling rule can cap `num_steps` proportionally to observed draft quality, preventing over-speculation at high BS Conceptually, the policy probes one step beyond the observed acceptance: ```text theme={null} target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps)) ``` So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down. ## Usage `--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode. ```bash theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-eagle-topk 1 \ --speculative-num-steps 3 \ --speculative-num-draft-tokens 4 \ --speculative-adaptive ``` If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`. Example config: ```json theme={null} { "ema_alpha": 0.2, "warmup_batches": 10, "update_interval": 5, "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0} } ``` Non-integer keys (`ema_alpha`, `warmup_batches`, `update_interval`) are global overrides applied to every BS slot. Integer keys (`"1"`, `"8"`) define per-BS slots. ## Config file reference The config file is optional. When provided, each integer BS-slot key must specify `candidate_steps`; all other keys fall back to defaults. ### Per-BS slot parameters
Key Default Meaning
candidate\_steps required Candidate speculative\_num\_steps tiers for this BS range. Must be a non-empty list of positive ints; a slot that omits it raises a config error
down\_hysteresis -0.25 Extra margin before moving to a smaller step
up\_hysteresis 0.0 Extra margin before moving to a larger step
ceiling\_coeff 0 (disabled) EMA ceiling coefficient; set > 0 to cap steps proportionally to draft quality
### Global parameters
Key Default Meaning
ema\_alpha 0.2 EMA smoothing factor for accepted draft length
update\_interval 5 Recompute interval, in verify batches, after warmup
warmup\_batches 10 Number of verify batches to observe before switching
## Monitoring You can inspect the active tier and acceptance metric via `/server_info`: ```bash theme={null} curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}' ``` * `speculative_num_steps` is the current active tier * `avg_spec_accept_length` helps explain whether the server is likely to move up or down ## Tuning tips * Start with the built-in default (conservative) — it is safe for all draft model qualities * For strong draft models, use the aggressive config with ceiling rule * Use fewer candidate steps if you want lower startup GPU memory overhead * Increase `ema_alpha` to react faster, or lower it for more stability * Increase `warmup_batches` or `update_interval` if tier switching is too noisy * At high batch sizes, narrower ladders (e.g., `[1, 2]` or `[1]`) often outperform wide ones * If your workload is already stable and one static setting is well tuned, adaptive mode may not help much ## Recommended configs The built-in default is conservative — safe for all draft models but may under-speculate for strong ones. Save one of these as a JSON file and pass via `--speculative-adaptive-config`. ### Conservative (default) — for weak draft models This is the built-in default: BS 8–31 allows `[1, 3]`, and BS≥32 locks to `step=1` to avoid wasted compute. Best for models like MiniMax-M2.5, DSV4. ```json theme={null} { "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0}, "32": {"candidate_steps": [1], "up_hysteresis": 0.0, "down_hysteresis": 0.0, "ceiling_coeff": 0} } ``` ### Aggressive — for strong or high-variance draft models Uses wider ladders with ceiling rule to cap speculation at high BS. Best for models like GLM-4.7-FP8. ```json theme={null} { "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0}, "8": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 3.0}, "64": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.67}, "128": {"candidate_steps": [1, 3], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.2} } ``` ### Custom per-model config For the best performance, benchmark your specific model across batch sizes with different static `num_steps` values, then build a per-BS config that matches each range's optimal step. A well-tuned per-model config might outperform the generic presets above. # Attention Backend Source: https://docs.sglang.io/docs/advanced_features/attention_backend SGLang supports a large variety of attention backends. Each of them has different pros and cons. You can test them according to your needs. Selecting an optimal attention backend is crucial for maximizing your performance. Different backends excel in various scenarios, so choose based on your model, hardware, and use case. Not all backends are supported on all platforms and model architectures. If you don't specify `--attention-backend`, SGLang makes a best effort to automatically select the most performant backend based on your hardware and model architecture. ## Support Matrix The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-4-mla-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434). ### MHA Backends
**Backend** **Page Size > 1 (native)** **FP8 KV Cache** **FP4 KV Cache** **Spec topk=1** **Spec topk>1** **Sliding Window** **MultiModal**
**FlashInfer** ✅ ✅ ❌ ✅ ✅ ✅ ❌
**FA3 (FlashAttention 3)** ✅ ✅ ❌ ✅ ✅ ✅ ✅
**FA4 (FlashAttention 4)** 128 ❌ ✅ ✅ ✅ ✅ ✅
**Triton** ❌ ✅ ✅ ✅ ✅ ✅ ✅
**Torch Native (SDPA)** ❌ ✅ ✅ ❌ ❌ ❌ ✅
**FlexAttention (PyTorch)** ❌ ❌ ✅ ❌ ❌ ❌ ❌
**TRTLLM MHA** 16, 32 or 64 ✅ ✅ ✅ ❌ ✅ ❌
**Dual Chunk FlashAttention** ✅ ❌ ❌ ❌ ❌ ❌ ❌
**HPC-Ops** 64 ✅ ❌ ❌ ❌ ❌ ❌
**AITER (ROCm)** ✅ ✅ ❌ ✅ ✅ ✅ ✅
**Wave (ROCm)** ✅ ❌ ❌ ❌ ❌ ❌ ❌
**Ascend (NPU)** ✅ ❌ ❌ ✅ ❌ ✅ ✅
**Intel XPU** ✅ ❌ ❌ ❌ ❌ ✅ ❌
**Intel AMX (CPU)** ❌ ❌ ❌ ❌ ❌ ❌ ❌
### MLA Backends
**Backend** **Native Page Sizes** **FP8 KV Cache** **FP4 KV Cache** **Chunked Prefix Cache** **Spec topk=1** **Spec topk>1**
**FlashInfer MLA** 1 ❌ ✅ ✅ ✅ ❌
**FlashMLA** 64 ✅ ✅ ✅ ✅ ❌
**Cutlass MLA** 128 ✅ ✅ ✅ ✅ ❌
**TRTLLM MLA (Blackwell)** 32 or 64 ✅ ✅ ✅ ✅ ❌
**CuteDSL MLA (Blackwell)** 32 or 64 ✅ ❌ ✅ ✅ ❌
**TokenSpeed MLA (Blackwell)** 32 or 64 ✅ (required) ❌ ✅ ✅ ❌
**FA3 (FlashAttention 3)** n/a ❌ ❌ ✅ ✅ ⚠️ (page\_size=1 only)
**Triton** n/a ❌ ❌ ❌ ✅ ⚠️ (page\_size=1 only)
**FA4** 1 ❌ ✅ ✅ ❌ ❌
**Ascend MLA (NPU)** 128 ❌ ❌ ❌ ❌ ❌
Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" column indicates whether a corresponding multimodal implementation exists for that backend family. * DSA is specifically designed for [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend](#dsa-attention-backend) section and [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) for details. For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm\_mha being incompatible with FA4, all other decode backends behave as shown in the table. Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths. Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical. Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128), HPC-Ops (64). MLA page-size constraints: * FlashInfer MLA: page\_size = 1. * FlashMLA: page\_size = 64. * Cutlass MLA: page\_size = 128. * TRTLLM MLA: page\_size ∈ \{32, 64}. * CuteDSL MLA: page\_size ∈ \{32, 64} (decode-only; prefill falls back to `trtllm_mla` when unset). * TokenSpeed MLA: page\_size ∈ \{32, 64} (Blackwell SM100/SM12x only; requires `--kv-cache-dtype fp8_e4m3`). ### GDN Attention Backends GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM). The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`. On SM100/SM103 with CUDA 13+, SGLang automatically selects FlashInfer for GDN prefill when the per-phase override is unset, the base linear-attention backend is Triton, recurrent state is BF16, key/value head dimensions are 128, dynamic chunking and page-major KV layout are disabled, and `--chunked-prefill-size` is between 1 and 8192. Radix caching may be disabled or use `no_buffer`, `extra_buffer`, or `extra_buffer_lazy`; the extra-buffer paths use state checkpoints.
Backend Decode Prefill / Extend Spec Decoding (Target Verify)
Triton (CUDA) ✅ ✅ ✅
Triton (AMD/ROCm) ✅ ✅ ✅
Triton (NPU) ✅ ✅ ❌
Triton (CPU) ✅ ✅ ❌
CuTe DSL (CUDA only) ✅ ❌ ❌
FlashInfer (CUDA, SM90/SM100/SM103) ✅ ✅ ✅ linear chain; tree falls back to Triton
GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models: * **Blackwell SM120 (e.g., RTX PRO 6000 Blackwell)**: `triton` or `flashinfer` for prefill/full attention; `trtllm_mha` is supported for `--decode-attention-backend` only. * **Other Blackwell variants (including SM100 B200/GB200)**: `triton`, `trtllm_mha`, or `fa4` only. * **NPU (Ascend)**: `ascend` only. * **AMD (ROCm)**: `triton` recommended. * **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints. ### DSA Attention Backend DSA (DeepSeek Sparse Attention) is a native sparse attention mechanism used by [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). It is activated automatically when the model architecture requires it and is selected via `--attention-backend dsa` (deprecated alias: `nsa`). Internally, the DSA backend dispatches to different sub-backends for prefill and decode phases. You can override these with `--dsa-prefill-backend` and `--dsa-decode-backend`:
Sub-backend Prefill Decode Notes
flashmla\_sparse ✅ ✅ Default prefill on Hopper and Blackwell (BF16)
flashmla\_sparse\_q8 ✅ ❌ Native FP8 (q8×kv8) sparse prefill on Hopper (SM90); requires --kv-cache-dtype fp8\_e4m3
flashmla\_kv ✅ ✅ Default for FP8 on Hopper (prefill + decode)
flashmla\_auto ✅ ❌ Picks flashmla\_sparse or flashmla\_kv by KV cache dtype
fa3 ✅ ✅ Default decode on Hopper (BF16)
trtllm ✅ ✅ Default decode on Blackwell (BF16); default for FP8 on Blackwell (prefill + decode)
tilelang ✅ ✅ Default on AMD (ROCm)
aiter ✅ ✅ AMD-specific kernel library (requires aiter package)
For deployment examples, see the [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2). ### Hybrid attention (different backends for prefill vs decode) (Experimental) Hybrid attention is an experimental feature. You can mix-and-match attention backends for prefill and decode. This is useful when one backend excels at prefill and another excels at decode. For the implementation details, please see `python/sglang/srt/layers/attention/hybrid_attn_backend.py`. ```bash Command theme={null} # Example: Prefill with FA4, Decode with TRTLLM MLA (Blackwell) python3 -m sglang.launch_server \ --model-path nvidia/DeepSeek-R1-FP4 \ --tp 8 \ --attention-backend trtllm_mla \ --moe-runner-backend flashinfer_trtllm \ --quantization modelopt_fp4 \ --prefill-attention-backend fa4 ``` #### Speculative decoding with hybrid attention Hybrid attention also works with speculative decoding. The backend used for draft decoding and target verification depends on `--speculative-attention-mode`: * `--speculative-attention-mode decode` (recommended): draft/verify use the decode backend. * `--speculative-attention-mode prefill` (default): draft/verify use the prefill backend. Constraints when combining hybrid attention with speculative decoding: * If any attention backend is `trtllm_mha`, speculative decoding supports only `--speculative-eagle-topk 1`. * For paged MHA backends with `--page-size > 1` and `--speculative-eagle-topk > 1`, only `flashinfer` is supported. * CUDA Graph: the decode backend is always captured; the prefill backend is captured only when `--speculative-attention-mode prefill`. If you set only one of `--prefill-attention-backend` or `--decode-attention-backend`, the unspecified phase inherits `--attention-backend`. If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase. ## Attention Backend Selection Guide (CUDA) If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture. ### Automatic Selection Logic **1. MHA Models (e.g., Llama, Qwen)** * **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported. * **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`. * **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`. **2. MLA Models (e.g., DeepSeek V3)** * **Hopper**: Defaults to `fa3` (requires CUDA 12.3+). * **Blackwell**: Defaults to `flashinfer`; `trtllm_mla` is auto-selected for DeepSeek V3 models specifically. * **Other Architectures**: Defaults to `triton`. ## User Guide ### Launch Command for Different Attention Backends * FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40) ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend flashinfer python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --attention-backend flashinfer \ --trust-remote-code ``` * FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20) ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend fa3 python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --trust-remote-code \ --attention-backend fa3 ``` * Triton ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend triton python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-V3 \ --attention-backend triton \ --trust-remote-code ``` * FlashMLA ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend flashmla \ --trust-remote-code python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend flashmla \ --kv-cache-dtype fp8_e4m3 \ --trust-remote-code ``` * TRTLLM MLA (Optimized for Blackwell Architecture, e.g., B200) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend trtllm_mla \ --trust-remote-code ``` * TRTLLM MLA with FP8 KV Cache (Higher concurrency, lower memory footprint) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend trtllm_mla \ --kv-cache-dtype fp8_e4m3 \ --trust-remote-code ``` * TRTLLM MHA (Optimized for Blackwell Architecture, e.g., B200) ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 4 \ --model Qwen/Qwen3.5-35B-A3B-FP8 \ --attention-backend trtllm_mha \ --trust-remote-code ``` * TRTLLM MHA (XQA backend) (Optimized for SM90 and SM120, e.g., H20, H200, 5090) Note that TRTLLM XQA backend only works well for pagesize 64. ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 4 \ --model Qwen/Qwen3.5-35B-A3B-FP8 \ --decode-attention-backend trtllm_mha \ --trust-remote-code ``` * HPC-Ops (MHA kernels from [HPC-Ops](https://github.com/Tencent/hpc-ops) by the Tencent Hunyuan AI Infra team; Hopper (SM90) only, requires installing the `hpc` package from source, page size 64, bf16 or fp8\_e4m3 KV cache, head\_dim 128, q/kv head group 4 or 8) ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --attention-backend hpc_ops \ --page-size 64 \ --trust-remote-code # FP8 models should also set --kv-cache-dtype fp8_e4m3 to run the FP8 attention # kernels. This enables the fused QKNorm+RoPE+FP8-quant+StoreKV op, which is # currently wired for Hunyuan V3 and requires per-rank (q_heads, kv_heads) of # (64, 8) or (8, 1), e.g. --tp 1 or --tp 8 for Hy3. python3 -m sglang.launch_server \ --tp 8 \ --model tencent/Hy3-FP8 \ --attention-backend hpc_ops \ --kv-cache-dtype fp8_e4m3 \ --page-size 64 \ --trust-remote-code ``` * FlashAttention 4 (MHA & MLA) ```bash Command theme={null} # FA4 for both prefill and decode on SM90/SM100 python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --attention-backend fa4 \ --page-size 128 \ --trust-remote-code python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --prefill-attention-backend fa4 \ --trust-remote-code ``` * Cutlass MLA ```bash Command theme={null} python3 -m sglang.launch_server \ --tp 8 \ --model deepseek-ai/DeepSeek-R1 \ --attention-backend cutlass_mla \ --trust-remote-code ``` * Ascend ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend ascend ``` * Intel XPU ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend intel_xpu ``` * Wave ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend wave ``` * FlexAttention ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend flex_attention ``` * Dual Chunk FlashAttention ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-14B-Instruct-1M \ --attention-backend dual_chunk_flash_attn ``` * Torch Native ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --attention-backend torch_native ``` ## Steps to add a new attention backend To add a new attention backend, you can learn from the existing backends (`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`) and follow the steps below. Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`. 1. Run without cuda graph. Support the two forward functions * forward\_extend * Will be used for prefill, prefill with KV cache, and target verification * It will be called once per layer * forward\_decode * Will be used for normal decode, and draft decode * It will be called once per layer * init\_forward\_metadata * Initialize the class and common metadata shared by all layers * Call the plan function for optimizations like split\_kv * It will be called once per forward 2. Run with cuda graph. It has two phases (capture and replay) and you need to implement three functions * init\_cuda\_graph\_state * It will be called once during life time * Create all common shared buffers * init\_forward\_metadata\_capture\_cuda\_graph * It will be called before capturing a cuda graph * It is similar to init\_forward\_metadata but write the medatada to some pre-defined buffers * init\_forward\_metadata\_replay\_cuda\_graph * It will be called before replaying a cuda graph * This function is in the critical path and needs to be fast # Breakable CUDA Graph Source: https://docs.sglang.io/docs/advanced_features/breakable_cuda_graph ## Motivation Standard CUDA graphs capture an entire forward pass as a single, opaque graph. This is great for performance, but creates two problems: 1. **Debugging is hard.** When something goes wrong inside a captured graph (wrong outputs, numerical mismatches, crashes), there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit. 2. **Some ops are incompatible.** Certain operations — dynamic control flow, host-device synchronization, JIT compilation, or ops that change behavior across iterations — cannot be captured into a CUDA graph at all. Today, the only workaround is to disable CUDA graphs entirely, which sacrifices the kernel launch overhead savings for the rest of the model. **Breakable CUDA Graph** solves both problems by allowing graph breaks to be inserted at specific points. The computation is split into multiple captured graph segments with eager (non-graph) execution in between. This preserves most of the CUDA graph performance benefit while allowing targeted operations to run outside the graph. ## Usage ### Debug Mode: Run Everything Eagerly The simplest use case is debugging. The `--debug-cuda-graph` flag wraps the entire decode forward pass in a graph break, so every operation runs eagerly while still going through the full CUDA graph capture/replay code path. This lets you debug CUDA graph issues without changing model code. ```bash theme={null} python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --debug-cuda-graph ``` This mode is intended for debugging only — it eliminates the performance benefit of CUDA graphs since every op runs eagerly. ### Selective Graph Breaks in Model Code For production use, you can mark specific functions as "non-graphable" using the `@eager_on_graph` decorator. During CUDA graph capture, these functions run eagerly between captured graph segments. Outside of capture, they behave normally. ```python theme={null} from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import eager_on_graph @eager_on_graph(enable=True) def my_dynamic_op(x): # This op is incompatible with CUDA graph capture return some_dynamic_operation(x) ``` You can also insert a bare graph break (no computation) using the `break_graph()` helper: ```python theme={null} from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import break_graph def forward(self, x): x = self.layer1(x) break_graph() # force a segment split here x = self.layer2(x) return x ``` To enable breakable CUDA graph at the environment level (without debug mode), set the environment variable: ```bash theme={null} export SGLANG_USE_BREAKABLE_CUDA_GRAPH=1 python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct ``` ### Server Args
Argument Default Description
--debug-cuda-graph False Enable debug/eager mode. Wraps the entire forward pass in a graph break so every op runs eagerly through the capture/replay path.
SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH 0 Environment variable. Enables breakable CUDA graph without debug mode. Required for @eager\_on\_graph decorators to take effect.
## How It Works ### Capture Breakable CUDA graph extends PyTorch's `torch.cuda.CUDAGraph` by splitting a single capture into multiple segments separated by graph breaks. During capture, the flow is: ``` Begin capture (segment 1) ... graphable ops ... @eager_on_graph function encountered: 1. End current capture segment 2. Run the function eagerly (allocates output tensors) 3. Record the function for later replay 4. Begin new capture segment ... more graphable ops ... End capture (segment N) ``` Each segment is independently instantiated as a CUDA graph executable. The non-graph functions and their argument references are stored for replay. ### Replay During replay: ``` For each segment i: 1. Launch CUDA graph segment i 2. Run the recorded non-graph function i eagerly Launch final CUDA graph segment ``` The non-graph functions are re-invoked with the same tensor references as capture time. Since these references point to the CUDA graph's static input/output buffers, they see updated values on each replay. ### Output Writeback When a non-graph function produces output during replay, the result must be written back into the same tensor buffers that downstream graph segments reference. The mechanism handles: * **Plain tensors**: In-place `copy_()` into the original buffer. * **Structured outputs** (dataclasses, objects with tensor attributes): Tensor fields are copied in-place; non-tensor fields are replaced. * **Dicts of tensors**: Tensor values are copied in-place; non-tensor values are replaced. ### Stream Fork/Join Tracking Some models fork work onto secondary CUDA streams (e.g., for overlapped computation). Breakable CUDA graph hooks `torch.cuda.Stream.wait_stream` to track which streams are forked from the capture stream. When a graph break occurs, all forked streams are automatically joined back before ending the capture segment, and re-forked after beginning the next segment. ## Compatibility * **CUDA and ROCm/HIP.** Breakable CUDA graph runs on both NVIDIA and AMD GPUs. Other platforms (NPU, CPU, MPS, XPU) are unsupported; there `--debug-cuda-graph` is automatically disabled with a warning. * **Requires `cuda-python` on NVIDIA.** Stream-capture-status queries use the CUDA runtime via `cuda.bindings` (`pip install cuda-python`); the portable `torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA. On ROCm/HIP — where `cuda-python` is unavailable — the portable `torch.cuda` API (which maps to the HIP runtime) is used instead. * **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`. ## Performance When no graph breaks are inserted, breakable CUDA graph has minimal overhead compared to standard CUDA graph — the capture/replay path is nearly identical. Each graph break adds: * One `cudaGraphLaunch` call (to replay the segment before the break) * One eager Python function call * One `cudaStreamBeginCapture` / `cudaStreamEndCapture` pair during capture For typical use cases with a small number of graph breaks, the overhead is negligible compared to the saved kernel launch overhead from the captured segments. ## Code Reference
File Description
python/sglang/srt/model\_executor/runner\_backend\_utils/breakable\_cuda\_graph/breakable\_cuda\_graph.py Core implementation: eager\_on\_graph, BreakableCUDAGraph, BreakableCUDAGraphCapture
python/sglang/srt/model\_executor/runner\_backend\_utils/breakable\_cuda\_graph/cuda\_utils.py CUDA runtime binding utilities (NVIDIA stream-capture queries)
python/sglang/srt/model\_executor/runner\_backend/breakable\_cuda\_graph\_backend.py Integration with CUDA graph runner backends
python/sglang/srt/server\_args.py --debug-cuda-graph flag and environment variable handling
python/sglang/srt/environ.py SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH environment variable definition
# Checkpoint Engine Integration Source: https://docs.sglang.io/docs/advanced_features/checkpoint_engine The SGLang checkpoint engine integration provides an efficient way to load model weights using a distributed checkpoint loading system. This feature significantly reduces model loading time, especially for large models and multi-node setups, by parallelizing the weight loading process across multiple processes and nodes. ## Overview The checkpoint engine integration allows SGLang to: * Load model weights in parallel using multiple processes * Distribute weight loading across multiple nodes to increase effective disk bandwidth * Overlap weight loading with other initialization tasks like CUDA graph capture * Support both single-node and multi-node deployments ## Installation First, install the checkpoint engine package: ```bash Command theme={null} pip install 'checkpoint-engine[p2p]' ``` ## Architecture The system consists of two main components: 1. **SGLang Server**: Runs with `--wait-for-initial-weights` flag to wait for weights before becoming ready 2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights The checkpoint engine uses a parameter server architecture with support for: * **Broadcast mode**: Weights are broadcast from loading processes to inference processes * **P2P mode**: Direct peer-to-peer weight transfer between processes * **All mode**: Combination of both broadcast and P2P methods ## Usage Examples ### Single Node Setup **Terminal 1 - Launch SGLang Server:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --wait-for-initial-weights ``` **Terminal 2 - Run Checkpoint Engine:** Using sglang entrypoint: ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` ### Multi-Node Setup (2 Nodes) **Node 0:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --wait-for-initial-weights \ --host [IP] ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 0 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` **Node 1:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --wait-for-initial-weights \ --host [IP] ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 1 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 8 ``` ### Multi-Node Setup with Tensor Parallelism (TP=16) **Node 0:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --wait-for-initial-weights \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ --node-rank 0 ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 0 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` **Node 1:** Launch SGLang server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ --wait-for-initial-weights \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ --node-rank 1 ``` Run checkpoint engine: Using sglang entrypoint (recommended): ```bash Command theme={null} python -m sglang.srt.checkpoint_engine.update \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` Using torchrun directly: ```bash Command theme={null} torchrun --nproc-per-node 8 \ --nnodes 2 \ --node-rank 1 \ --master-addr [IP] \ --master-port 29500 \ examples/checkpoint_engine/update.py \ --update-method broadcast \ --checkpoint-path /path/to/Qwen/Qwen3-8B/ \ --inference-parallel-size 16 ``` ## Configuration Options ### SGLang Server Options * `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks) * `--wait-for-initial-weights`: Wait for checkpoint engine to provide weights before becoming ready * `--host`: Host address for multi-node setups * `--dist-init-addr`: Distributed initialization address for tensor parallelism ### Checkpoint Engine Options * `--update-method`: Weight update method (`broadcast`, `p2p`, or `all`) * `--checkpoint-path`: Path to model checkpoint directory * `--inference-parallel-size`: Number of inference parallel processes * `--endpoint`: SGLang server endpoint (default: `http://localhost:19730`) * `--checkpoint-name`: Name for the checkpoint (default: `my-checkpoint-iter-0`) * `--save-metas-file`: File to save checkpoint metadata * `--load-metas-file`: File to load checkpoint metadata from * `--uds`: Unix domain socket path for communication * `--weight-version`: Version identifier for weights ## Performance Benefits The checkpoint engine provides significant time savings in two main aspects: 1. **Multi-node Loading**: Each node only loads a portion of weights from disk, effectively increasing disk bandwidth. More participating nodes provide greater acceleration. Preliminary tests show 20-second acceleration when loading DeepSeek-R1 on H20-3e with two nodes. 2. **Single Process Optimization**: Using dummy format allows overlapping disk-to-CPU transfer with CUDA graph capture and other initialization tasks, providing additional time savings. ## Troubleshooting * Ensure checkpoint engine package is installed: `pip install 'checkpoint-engine[p2p]'` * Verify network connectivity between nodes in multi-node setups * Check that the checkpoint path contains valid model files * Monitor logs for connection errors between SGLang server and checkpoint engine * Use `--sleep-time` parameter to add delays if needed for debugging ## References * [Checkpoint Engine Repository](https://github.com/MoonshotAI/checkpoint-engine) # Cuda Graph for Multi-Modal Encoder in SGLang Source: https://docs.sglang.io/docs/advanced_features/cuda_graph_for_multi_modal_encoder ## Motivation In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits: Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches. Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency. Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes). The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact. This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance. ## Design and Restrictions The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details. ### Dynamic inputs to fit static constraints of CUDA Graph Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph\_key = S). The first time create a new S, and then capture a graph; afterwards, replay it. If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs. ### Stable addresses Everything "parameter-like" becomes a static buffer: * block\_input / block\_ws / block\_output * cu\_full\_len / cu\_window\_len and their kk variants * sin\_cos\_ws In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents. ### Attention backend arguments Attention backend arguments are fixed inside the graph: TritonAttn expects \[cu\_seqlens, cu\_seqlens\_kk, max\_len] FA3 expects \[cu\_seqlens, max\_len] max\_len is frozen as an int constant. cu\_seqlens is cached into a dict during create\_graph(), and its contents are not updated during subsequent replays. For the same graph\_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu\_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly. ### Rotary buffer management The feature reallocates a larger sin\_cos\_ws when seq\_len increases. The max\_content\_len is used to make sure the maximum size of the allocated rotary buffer. ## Command Example You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example: ```shell Command theme={null} SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct ``` Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example: ```shell Command theme={null} SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct \ --piecewise-cuda-graph-max-tokens 4096 \ --enable-piecewise-cuda-graph \ --piecewise-cuda-graph-compiler eager ``` ## Known supported models * Qwen2.5-VL ([https://github.com/sgl-project/sglang/pull/14422](https://github.com/sgl-project/sglang/pull/14422)) * Qwen3-VL ([https://github.com/sgl-project/sglang/pull/15320](https://github.com/sgl-project/sglang/pull/15320)) # Decode Context Parallelism Source: https://docs.sglang.io/docs/advanced_features/dcp Decode context parallelism (DCP) stripes a request's MLA KV cache across ranks in an existing tensor-parallel group. With DCP size `c`, rank `r` owns position `p` when `p mod c = r`, so each rank stores and reads roughly `1/c` of the cache. Local attention then sees only part of the context. The MLA path returns a partial output and log-sum-exp (LSE), exchanges both in one packed all-to-all, and merges them exactly. DCP complements TP and [attention data parallelism (DPA)](/docs/advanced_features/dp_dpa_smg_guide) rather than replacing them. The primary path is absorbed MLA decode and static target verification. `dcp_size=1` retains existing non-DCP behavior. On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchanged, so DCP grows long-context KV capacity but does not raise the KDA concurrency ceiling. ## Enable DCP | Setting | Behavior | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | | `--dcp-size N` | Enables DCP and widens virtual capacity and page size by `N`. Alias: `--decode-context-parallel-size`. | | `--dcp-comm-backend` | Selects the MLA partial-output merge: `ag_rs`, `a2a`, or `fi_a2a`. | | `--dcp-replicate-q-proj` | Removes the query all-gather for supported MLA weights. Use `--no-dcp-replicate-q-proj` to disable a model-specific default. | | `--enable-dp-attention` | Composes only when DCP groups nest inside attention TP. | A DCP group must fit inside one attention-TP group and one attention-DP replica: ```text Topology theme={null} attn_tp_size = tp_size / attn_dp_size attn_tp_size % dcp_size == 0 ``` For example, `TP=64, DP=4, DCP=16` creates four valid 16-rank attention replicas. `DCP=32` would cross replica boundaries and is invalid. Startup currently checks only `tp_size % dcp_size == 0`, not the stronger condition above. Until that check is fixed, DPA+DCP deployments must enforce containment when choosing the topology. MLA decode on DeepSeek-V3.1 with DCP 8 inside TP 8: ```bash Command theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.1 \ --trust-remote-code \ --tp-size 8 \ --dcp-size 8 \ --host 0.0.0.0 \ --port 30000 ``` Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default and picks `fi_a2a` on MNNVL systems, otherwise `a2a`: ```bash Command theme={null} sglang serve \ --trust-remote-code \ --model-path moonshotai/Kimi-K3 \ --tp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --dp-size 4 \ --dcp-size 8 \ --host 0.0.0.0 \ --port 30000 ``` See the [Kimi K3 cookbook](/cookbook/autoregressive/Moonshotai/Kimi-K3) for large-scale presets that combine DCP with DPA and expert parallelism. ## Why decode context parallelism? Absorbed MLA shares one latent KV representation across all query heads. TP can split the heads and weights, but not that cache, so every attention-TP rank stores and reads the same context. ```mermaid theme={null} flowchart LR subgraph TP["TP only: every rank stores the full MLA context"] direction TB T0["Rank 0
tokens 0 1 2 3 4 5 6 7"] T1["Rank 1
tokens 0 1 2 3 4 5 6 7"] T2["Rank 2
tokens 0 1 2 3 4 5 6 7"] T3["Rank 3
tokens 0 1 2 3 4 5 6 7"] end TP -- "stripe by position" --> DCP subgraph DCP["DCP4: one interleaved shard per rank"] direction TB D0["Rank 0
tokens 0 4"] D1["Rank 1
tokens 1 5"] D2["Rank 2
tokens 2 6"] D3["Rank 3
tokens 3 7"] end ``` DCP stripes that cache round-robin by logical token position, keeps a shared virtual index on every rank, runs attention against the owner-local shard, and merges partial outputs by LSE into the usual TP-local head layout. ## Architecture ### Virtual KV layout Let each rank have physical token capacity `C` and physical page size `P`. DCP exposes a shared virtual allocator: ```text Layout theme={null} virtual capacity = C * c virtual page size = P * c owner(v) = v mod c physical(v) = floor(v / c) ``` Every rank sees the same request-to-token map. On write, it keeps its positions and compacts them into the local physical pool. One widened virtual page maps to one physical page on each rank, so the layout stays balanced to within one token as a sequence grows. ### MLA decode dataflow ```mermaid theme={null} flowchart LR H["Hidden states"] --> Q["Full DCP-group query heads"] Q --> A0["Rank 0 local attention
partial O0 + LSE0"] Q --> A1["Rank 1 local attention
partial O1 + LSE1"] Q --> AN["Rank c-1 local attention
partial O + LSE"] A0 --> A2A["Packed all-to-all"] A1 --> A2A AN --> A2A A2A --> MERGE["Exact LSE merge"] MERGE --> OUT["TP-local head shard"] ``` For context partition `r`, the kernel returns locally normalized output `o_r` and `lse_r`. The global result is `lse = logsumexp_r(lse_r)` and `o = sum_r(exp(lse_r - lse) * o_r)`. This is exact aside from floating-point reduction order. The merge uses the attention backend's LSE base, either base-e or base-2. ### Query-projection replication Normally each layer all-gathers its absorbed and rotary query components. `--dcp-replicate-q-proj` gathers the query-projection and `w_kc` weights once at startup and computes the full DCP-group query locally. The trade is more weight memory and GEMM work for one fewer collective per layer. Only unquantized BF16/FP16 weights use this path; other layers fall back to the query all-gather. ### Communication backends | Backend | DCP collectives per MLA layer | Notes | | ---------------------- | ---------------------------------- | -------------------------------------- | | `ag_rs` | Query AG + LSE AG + FP32 output RS | Generic fallback | | `a2a`, gathered Q | Query AG + packed NCCL A2A | Two collectives | | `a2a`, replicated Q | One packed NCCL A2A | Canonical low-latency path | | `fi_a2a`, replicated Q | One FlashInfer MNNVL A2A | Requires CUDA, SM90+, and MNNVL fabric | Kimi K3 enables replicated Q by default and picks `fi_a2a` on MNNVL systems, otherwise `a2a`. Its decode backend is `cutedsl_mla`. The generic defaults remain `ag_rs` and model-resolved query replication. DCP adds a context-independent decode collective while cutting context-dependent KV storage and reads by about `c`. Extend is outside that decode cost model: it may gather cached prefix shards, restore token order, and append new tokens, and that work grows with prefix length. ## Compositions ### DCP × speculative decoding The draft KV cache is replicated: every DCP rank stores the full draft context. DCP only stripes the target MLA KV, so it saves target cache, not draft cache. Target verify and decode use the DCP-aware MLA kernel `cutedsl_mla`. It: 1. Builds a rank-local page table from the cyclic token map (`owner(p) = p mod c`). 2. Runs attention against that shard, passing `cp_world`, `cp_rank`, and the global KV lengths so causality still sees the full sequence. 3. Returns a rank-local `(output, LSE)` that the usual DCP all-to-all merge combines. Draft steps use the same backend against the replicated draft cache, so they skip DCP metadata. Kimi K3 DCP selects `cutedsl_mla` automatically. ### DCP × PD disaggregation A dense prefill page cannot be copied directly into striped decode storage. The PD sender uses the decode rank's DCP metadata to select and pack the right rows. See [PD disaggregation](/docs/advanced_features/pd_disaggregation) for the transfer engines. * Equal DCP sizes require matching DCP ranks and use the existing page path. * DCP1 prefill to DCP decode uses token relayout for MLA and hybrid-MLA pools. * All other DCP-size transitions are rejected. The plan includes the exact token count, so stale rows do not leak out of a partial final page. Mooncake and NIXL use the same plan. KDA state keeps its attention-TP mapping and bypasses DCP filtering. This composition requires Mooncake or NIXL, matching physical page size and KV dtype, prefill attention CP `1`, and decode chunk cache. Decode radix cache and HiCache are not supported here. ### DCP × HiCache L2 HiCache keeps one widened logical page space across L1 and L2. The controller sees `P * c` slots, while each GPU and host buffer stores only its `P` local rows. See [HiCache](/docs/advanced_features/hicache) for the cache hierarchy. Before H2D or D2H, both index lists keep positions owned by this rank and map `i` to `floor(i / c)`. Transfers cover whole widened pages, and every rank moves its shard independently, with no DCP collective. For Kimi K3 this applies to MLA KV; KDA/Mamba state uses its own request-indexed host pool. The current scope is MLA L1/L2. L3, LMCache, HiSparse, non-MLA KV host pools, speculative decoding, and PD decode are not supported in this combination. ## References * [DCP and Helix Parallelism roadmap](https://github.com/sgl-project/sglang/issues/29736) * [SGLang and Miles Add Day-0 Support for Kimi K3](https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support#decode-context-parallelism) * [Original SGLang DCP design](https://docs.google.com/document/d/1pjBSID0palpuI7rx8Ck7Z0BcyzBXAgnkDrf9uQg0uqU/edit?tab=t.0) # Deterministic Inference Source: https://docs.sglang.io/docs/advanced_features/deterministic_inference ## Why Deterministic Inference Matters Deterministic inference ensures consistent LLM outputs across runs, which is critical for: * **Reinforcement Learning**: Ensures consistent logprobs across runs, reducing stochastic noise and making RL training more stable, reproducible, and debuggable. * **Testing & Debugging**: Enables reproducible validation * **Production**: Improves reliability and user experience Even with `temperature=0`, standard LLM inference can produce different outputs due to dynamic batching and varying reduction orders in GPU kernels. ## The Root Cause of Non-Determinism The main source is **varying batch sizes**. Different batch sizes cause GPU kernels to split reduction operations differently, leading to different addition orders. Due to floating-point non-associativity (`(a + b) + c ≠ a + (b + c)`), this produces different results even for identical inputs. ## SGLang's Solution Building on [Thinking Machines Lab's batch-invariant operators](https://github.com/thinking-machines-lab/batch_invariant_ops), SGLang achieves fully deterministic inference while maintaining compatibility with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling. The development roadmap for deterministic inference features can be found in this [issue](https://github.com/sgl-project/sglang/issues/10278). ### Supported Backends Deterministic inference is only supported with the following three attention backends: **FlashInfer**, **FlashAttention 3 (FA3)**, and **Triton**. The following table shows feature compatibility for deterministic inference across different attention backends:
Attention Backend CUDA Graph Chunked Prefill Radix Cache Non-greedy Sampling (Temp > 0)
**FlashInfer** ✅ Yes ✅ Yes ❌ No ✅ Yes
**FlashAttention 3 (FA3)** ✅ Yes ✅ Yes ✅ Yes ✅ Yes
**Triton** ✅ Yes ✅ Yes ✅ Yes ✅ Yes
## Usage ### Basic Usage Enable deterministic inference by adding the `--enable-deterministic-inference` flag: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --attention-backend fa3 \ --enable-deterministic-inference ``` ### Server Arguments
Argument Type/Default Description
`--enable-deterministic-inference` flag; default: disabled Enable deterministic inference with batch-invariant operations
`--attention-backend` string; default: fa3 Choose attention backend (flashinfer, fa3, or triton)
### Example Configurations #### Qwen3-8B ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --attention-backend flashinfer \ --enable-deterministic-inference ``` #### Llama Models ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --attention-backend fa3 \ --enable-deterministic-inference ``` #### Qwen3-30B-A3B (MoE Model) ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B \ --attention-backend fa3 \ --enable-deterministic-inference ``` ### Deterministic Inference with Non-Greedy Sampling (Temperature > 0) SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds. This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative Policy Optimization) where you need multiple diverse but reproducible responses. #### Default Behavior By default, SGLang uses a sampling seed of `42` for reproducible sampling: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "Tell me a joke", "sampling_params": { "temperature": 0.8, # Non-greedy sampling "max_new_tokens": 128, }, }, ) print(response.json()) # This will always produce the same response across runs ``` #### Generating Multiple Reproducible Responses To sample different responses from the same prompt while maintaining reproducibility (e.g., for GRPO training), provide different sampling seeds in your requests: ```python Example theme={null} import requests # Prepare a list of sampling seeds for different responses sampling_seeds = [42, 43, 44, 45, 46] responses = [] for seed in sampling_seeds: response = requests.post( "http://localhost:30000/generate", json={ "text": "Tell me a joke", "sampling_params": { "temperature": 0.8, "max_new_tokens": 128, "sampling_seed": seed, # Specify sampling seed }, }, ) responses.append(response.json()) # Each seed will produce a different but reproducible response # Using the same seed will always produce the same response ``` This approach ensures that: * Different seeds produce diverse responses * The same seed always produces the same response across different runs * Results are reproducible for debugging and evaluation ## Verification Run deterministic tests to verify consistent outputs: ```bash Command theme={null} # Single test: same prompt, varying batch sizes python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50 # Prefix test: prompts with different prefix lengths python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50 # Radix Cache Consistency mode: test radix cache determinism (cached vs uncached prefill) python3 -m sglang.test.test_deterministic --test-mode radix_cache ``` Expected result: All tests should show `Unique samples: 1` (perfectly deterministic). # DP, DPA and SGLang DP Router Source: https://docs.sglang.io/docs/advanced_features/dp_dpa_smg_guide This guide explains the difference between Data Parallelism (DP) and Data Parallelism Attention (DPA), how to enable each mode correctly, and how to use the SGLang Model Gateway (SMG) for production-grade DP deployments. ## Data Parallelism (DP) **Data Parallelism (DP)** is the most common parallelism strategy that replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. Each GPU set handles independent requests. With dedicated routing strategies, as we will introduce later, with those proper routing algorithms in SGLang Model Gateway, the throughput of your serving system could be multiplied nearly linearly. ### Key characteristics * Each replica has a full copy of the model * Requests are distributed/scattered across replicas * No inter-replica communication during one request's inference (for simple DP) ## Data Parallelism Attention (DPA) **Data Parallelism Attention (DPA)**, also known as DP Attention, is an advanced parallelism strategy. While DPA provides the most significant benefits for **Multi-Head Latent Attention (MLA)** models (such as DeepSeek, MiniMax, Kimi-K2), it also supports **standard attention models** like Qwen. ### The Problem with Tensor Parallelism for MLA Models The most common parallelism strategy for inference is **Tensor Parallelism (TP)**. However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have **one KV head**. If we use tensor parallelism on 8 GPUs, it will lead to: * **Duplicated KV cache** across all GPUs * **Unwanted memory usage** that limits batch size * **Reduced throughput** due to memory constraints ### How DPA Works DPA addresses these limitations by applying **data parallelism specifically to the attention component**.
DPA + EP Architecture

Each DP replica:

  • Processes different batches independently (can be in different forward modes: prefill, decode, or idle)
  • Maintains its own KV cache (no duplication)
  • Enables significantly larger batch sizes due to memory savings

Communication patterns in DPA + EP:

  • All2All (Dispatch): Routes tokens to expert sub-groups based on gating decisions
  • All2All (Combine): Gathers computed results from experts back to original token positions
### Key benefits of DPA 1. **Significantly reduced KV cache memory**: Each DP replica only stores KV cache for its own batches 2. **Larger batch sizes**: Memory savings enable larger batch sizes 3. **Improved decoding throughput**: Significant throughput gains for MLA-based models 4. **Independent forward modes**: Each DP replica can be in different forward modes (prefill, decode, or idle) and handles its assigned batches independently during attention computation ### DPA with Expert Parallelism for MoE For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (EP) for best throughput at scale. However, **DPA does not require EP**: you can enable DPA without EP if your deployment does not need expert sharding. * Distribute 256+ expert weights across GPUs (cannot fit on a single GPU) * Enable efficient all-to-all token routing via DeepEP * Scale to large clusters (up to 5x throughput improvement over vanilla TP) ### Recommended setup for DeepSeek ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --dp-size 8 \ --ep 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --moe-runner-backend deep_gemm ``` > **Note**: `--dp-size` must be explicitly set when using `--enable-dp-attention`. If `dp_size` is 1 (default), DPA will be disabled. For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism). ### Target Models DPA supports the following model architectures: * **MLA (Multi-Head Latent Attention) models** - where DPA provides the most significant benefits: * DeepSeek family (DeepSeek-V2, DeepSeek-V3, DeepSeek-R1) * MiniMax models * Kimi-K2 * Other models using MLA architecture * **Standard attention models** - also supported: * Qwen models (see [PR #6121](https://github.com/sgl-project/sglang/pull/6121)) For models like Llama, with standard GQA, standard DP, or TP is typically recommended. To enable DPA, add `--enable-dp-attention` to your server launch command. ### Activation Logic DPA is enabled explicitly via server arguments (CLI or config). You must set both `--dp-size` and `--enable-dp-attention`: ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ --tp 8 \ --dp-size 8 \ --enable-dp-attention ``` **Important**: `--dp-size` must be greater than 1 for DPA to work. When `dp_size == 1` (default), `--enable-dp-attention` is automatically disabled. The constraint `tp_size % dp_size == 0` must also be satisfied. ### TP LMHead with all-to-all `dp-lm-head` keeps full weight and computes the local batch, making the gemm inefficient when batchsize per-rank is small. The existing `tp-lm-head` improves the gemm efficiency but introduce heavy allgather communication on full vocabulary. Therefore, all-to-all can be applied to eliminate the heavy allgather while retaining the gemm efficiency under small per-rank batchsize. Each rank will send and receive only the local rows with sharded vocab, which is more lightweight. This optimization is **enabled by default for decode node and pure DPA**. Users can use `--no-enable-tp-lm-head-all-to-all` to opt out, or `--enable-tp-lm-head-all-to-all` for force enabling. ```bash theme={null} python -m sglang.launch_server \ --model-path MODEL_PATH \ --disaggregation-mode decode \ --tp 8 \ --dp-size 8 \ --enable-dp-attention ``` Prefill / colocated workers will fallback to TP LMHead with allgather, as well as other parallel strategies. More details can be found in [PR #32313](https://github.com/sgl-project/sglang/pull/32313). ### Standard DP for MLA models Note that MLA models, of course, also support DP. Suppose you want to enable standard DP for MLA models. First, launch each MLA model's replica independently. You may launch these replicas one by one with DPA enabled. After launching each MLA model's replica, launch an SMG and connect all the replicas to the SMG. A detailed explanation of SMG is as follows. ## Modern Data Parallelism SGLang Model Gateway (SMG) ### Native DP Mode Native DP (built-in Data Parallelism) in SGLang creates multiple worker processes within a single SGLang instance, under the control of `DataParallelController` with the launching parameter of `dp-size`. ```bash theme={null} # Native DP mode python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 ``` **Limitations:** * Built-in in-process load balancing only (e.g., `round_robin`, `total_requests`, `total_tokens`) * No cache-aware routing * Limited observability and metrics * No fault tolerance or circuit breakers * Not suitable for production workloads ⚠️ Native DP is **highly not recommended for use right now**. It is only used in some ancient/outdated RL frameworks. You can use SGLang Model Gateway (SMG) to power up your data parallelism in any use case. ### SMG-Based DP (Recommended) Starting from September 2024, SGLang Model Gateway, i.e., SMG, formerly named as SGLang DP Router, was built especially as a production-ready DP routing system with Rust. It starts from DP routing, but later we further expanded its scope to coordinate RL, PD Disaggregation, and other scenarios. This doc only discusses SMG's usage in DP routing. For other usage, please refer to [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). > To achieve the best production-level routing performance and reduce the overhead to an extreme extent, we use Rust to build SMG, but not Python, since Python is never FAST enough. **We strongly recommend using the SGLang Model Gateway (SMG) for production-grade Data Parallelism.** SMG provides significant advantages over native DP mode. ```bash theme={null} # SMG-based DP mode (Recommended) python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 ``` ⚠️ Note that **SMG and Naive DP share the same launching parameter, `--dp-size`**. But the entrypoint of Naive DP is `python -m sglang.launch_server`, and SMG's entrypoint is `python -m sglang_router.launch_server`. **Advantages of SMG-Based DP:**
Feature Native DP SMG-Based DP
Load Balancing Built-in in-process methods Advanced policies (cache-aware, power-of-two, etc.)
Cache Awareness ❌ No ✅ Yes - significantly higher cache hit rate
Throughput Baseline Significant improvement
Multi-Node Support Limited ✅ Full support
Worker Health Monitoring Basic ✅ Circuit breakers, health checks
Reliability Basic ✅ Retries, rate limiting, queuing
Observability Basic metrics ✅ 40+ Prometheus metrics, OpenTelemetry
Hot Worker Add/Remove ❌ No ✅ Yes
### SMG's Performance The cache-aware routing policy in SMG significantly improves performance for workloads with shared prefixes:
Metric Without Cache-Aware With Cache-Aware SMG
Throughput (token/s) 82,665 158,596 (+92%)
Cache Hit Rate 20% 75% (+275%)
*Benchmark from [SGLang v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), workload with multiple long prefix groups, 8x A100 80GB GPUs, dp-size=8* ### When to Use Each **Use Native DP when:** * ~~Never use Native/Naive DP~~ * Learning material of DP routing **Use SMG-Based DP when:** * In any case, when you think DP is needed * Production deployments * Multi-node distributed setups * Workloads with shared prefixes (high cache reuse potential) * You need high availability and reliability features * You require detailed observability and metrics * You want to have highly efficient RL rollout systems Note that for RL rollout systems, **there are four crucial reasons that SMG-Based DP is far better than naive DP routing**. Details can be found at [Load Balancing Router in RL](/docs/advanced_features/sglang_for_rl#load-balancing-router). ### Quick Start For SMG **Installation** ```bash theme={null} pip install sglang-router # or pip install "sglang[all]" ``` **Option A: Co-launch Workers and SMG (Simplest)** This is the easiest way to get started - SMG and workers are launched together: ```bash theme={null} python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --host 0.0.0.0 \ --port 30000 ``` **Option B: Separate Launch (Multi-Node)** For distributed deployments across multiple machines: 1. Launch workers on each node ```bash theme={null} # Node 1 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 8000 # Node 2 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 8000 ``` 2. Launch SMG pointing to workers ```bash theme={null} python -m sglang_router.launch_router \ --worker-urls http://node1:8000 http://node2:8000 \ --policy cache_aware \ --host 0.0.0.0 \ --port 30000 ``` **Option C: Dynamic Worker Registration** For elastic deployments where workers can be added/removed dynamically: ```bash theme={null} # Launch SMG first python -m sglang_router.launch_router \ --policy cache_aware \ --host 0.0.0.0 \ --port 30000 # Register workers dynamically curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url": "http://worker1:8000"}' curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url": "http://worker2:8000"}' ``` ### Load Balancing Policies SMG supports multiple load balancing policies:
Policy Description Best For
cache\_aware Combines cache locality with load balancing Recommended for most workloads
round\_robin Cycles through workers in order Simple, predictable distribution
random Random worker selection Baseline, testing
power\_of\_two Samples two workers, picks lighter one Low latency requirements
**Cache-Aware Policy (Default, Recommended)** The cache-aware policy provides the best performance for most workloads: ```bash theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware \ --cache-threshold 0.5 \ --balance-abs-threshold 32 \ --balance-rel-threshold 1.5 \ --eviction-interval-secs 120 \ --max-tree-size 67108864 ``` **How it works:** 1. Maintains an approximate radix tree for each worker based on request history 2. Routes requests to workers with the highest prefix match (cache hit) 3. Falls back to shortest-queue routing when load is imbalanced 4. Automatically evicts old entries to prevent memory overflow ### Best Practices 1. **Start with `cache_aware` policy** - It provides the best balance between cache locality and load distribution for most workloads 2. **Use SMG for production** - Prefer `sglang_router.launch_server` over `sglang.launch_server` for better reliability and observability 3. **Enable health checks** - Configure `--router-health-check-interval-secs` to detect and remove unhealthy workers automatically **Recommended command with best practices applied:** ```bash theme={null} python -m sglang_router.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --router-policy cache_aware \ --router-health-check-interval-secs 30 \ --router-prometheus-port 10001 \ --host 0.0.0.0 \ --port 30000 ``` For advanced configuration (circuit breakers, retries, Prometheus metrics, K8s integration), see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). ### Verifying Traffic Distribution After launching SMG, verify that traffic is being distributed correctly: **1. Check worker status:** ```bash theme={null} curl http://localhost:30000/workers ``` **2. Check load distribution:** ```bash theme={null} curl http://localhost:30000/get_loads ``` **3. Monitor metrics (if Prometheus enabled):** ```bash theme={null} # Key metrics to check smg_router_requests_total{model="..."} smg_worker_requests_active{worker="..."} sglang_cache_hit_rate{source="..."} ``` For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway). ## Reference
Strategy Use Case Key Benefit
Native DP (--dp-size) Never Easy to understand, not rust based
SMG-Based DP Production (recommended) Cache-aware routing, high availability
DPA (--dp-size N --enable-dp-attention) DeepSeek/MLA models Eliminates KV cache duplication, improved throughput
DPA + EP DeepSeek MoE models Significant throughput improvement vs vanilla TP
**Recommended production setup for DeepSeek:** 1. Enable **DPA** for attention layers (`--dp-size 8 --enable-dp-attention`) 2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`) 3. Use **SMG** with **cache\_aware** policy **Related documentation:** * [Expert Parallelism](./expert_parallelism) - DeepEP, Two-Batch Overlap, EPLB * [SGLang Model Gateway Documentation](./sgl_model_gateway) - SMG configuration & troubleshooting * [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - 96 GPU deployment guide # DP for Multi-Modal Encoder in SGLang Source: https://docs.sglang.io/docs/advanced_features/dp_for_multi_modal_encoder A typical VLM architecture involves two main components: an multi-modal encoder and a text decoder. Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model. The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features. However, since the size of ViT is very small compared to language decoders, there is relatively little gain from TP. On the other hand, TP incurs significant communication overhead because of all-reduce being performed after every layer. Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput. In this hybrid layout, the vision front-end becomes parallel and lightweight, while scarce interconnect bandwidth and collective ops are reserved for the LLM. Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. ## Command Example You can enable batch-level DP by setting `mm-enable-dp-encoder`, for example: ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen2.5-VL-7B-Instruct \ --tp 2 \ --mm-enable-dp-encoder ``` ## Known supported models * Qwen2.5-VL (\<[https://github.com/sgl-project/sglang/pull/13126](https://github.com/sgl-project/sglang/pull/13126)>) * Qwen3-VL (\<[https://github.com/sgl-project/sglang/pull/13724](https://github.com/sgl-project/sglang/pull/13724)>) * InternVL (\<[https://github.com/sgl-project/sglang/pull/13925](https://github.com/sgl-project/sglang/pull/13925)>) * GLM-4.5V & GLM-4.6V (\<[https://github.com/sgl-project/sglang/pull/14097](https://github.com/sgl-project/sglang/pull/14097)>) # EPD Disaggregation Source: https://docs.sglang.io/docs/advanced_features/epd_disaggregation ## Why and What is EPD Disaggregation? In modern Vision-Language Model (VLM) inference, request execution naturally decomposes into three distinct stages: Encoder, Prefill, and Decode. The Encoder stage performs vision preprocessing and ViT-based image encoding, which is highly compute-intensive but only required during request initialization. The Prefill stage processes the full multimodal input sequence to initialize the language model’s Key-Value (KV) cache, while the Decode stage is dominated by memory bandwidth and KV cache access for autoregressive token generation. Existing deployments typically colocate these stages within a unified execution engine, or at best apply Prefill–Decode (PD) disaggregation. However, such designs still tightly couple vision encoding with language prefill, leading to inefficient resource utilization, limited scalability for image-heavy workloads, and suboptimal scheduling under load. To address these challenges, we introduce Encoder–Prefill–Decode (EPD) Disaggregation in SGLang. EPD further separates vision encoding from language processing, enabling independent horizontal scaling of encoder servers, improved load balancing for multimodal requests, and seamless integration with existing PD disaggregation to form a fully decoupled three-tier inference architecture. ### Usage You can launch a language-only model using `--language-only`, or an encoder-only model using `--encoder-only`. When launching a language-only model, you must additionally specify the encoder service endpoints via `--encoder-urls`. We support multiple encoder transfer backends, including zmq\_to\_scheduler, zmq\_to\_tokenizer, and mooncake (the default is zmq\_to\_scheduler). The backend can be selected using `--encoder-transfer-backend`. ### Encoder transfer with Mooncake `--encoder-transfer-backend mooncake` controls **how encoder outputs are transferred** between encoder and language/prefill services. It is an encoder transfer option and can be used independently of the global multimodal embedding cache. Example: ```bash Command theme={null} # encoder python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend mooncake \ --port 30000 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 \ --encoder-transfer-backend mooncake \ --port 30002 ``` ### Global multimodal embedding cache with Mooncake SGLang also supports a Mooncake-backed **global multimodal embedding cache** for EPD workloads. When enabled on encoder servers, repeated image inputs can reuse previously computed ViT embeddings across instances instead of running the vision encoder again. This feature is useful when: * the deployment serves repeated or overlapping image inputs, * encoder compute is the bottleneck, and * Mooncake is already available in the cluster. At a high level, the encoder checks whether the image embedding already exists in Mooncake. Cache hits are prefetched from the global store, while misses are encoded normally and inserted into the cache in the background. To enable it: * install and configure Mooncake in the same way as other SGLang Mooncake integrations, * add `--enable-mm-global-cache` on the encoder server. `--enable-mm-global-cache` controls **whether multimodal embeddings are looked up and stored in the global Mooncake cache**. It is separate from `--encoder-transfer-backend`, which only controls encoder output transport. For Mooncake deployment and configuration details, see [HiCache best practices](./hicache_best_practices#deployment-with-mooncake) and the [Mooncake backend README](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md). Example: ```bash Command theme={null} # Shared Mooncake configuration export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" export MOONCAKE_MASTER="127.0.0.1:50051" export MOONCAKE_PROTOCOL="rdma" export MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb" # encoder with global multimodal cache enabled python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --enable-mm-global-cache \ --port 30000 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 \ --port 30002 ``` Notes: * This cache is for **multimodal encoder embeddings**, not the language model KV cache. * The feature currently uses Mooncake as the shared backing store. * It can be enabled regardless of which `--encoder-transfer-backend` you use. * It is most relevant for EPD or encoder-disaggregated VLM deployments where the same images are likely to appear across requests or instances. #### Qwen VL * EP Disaggregation ```bash Command theme={null} # encoder 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # encoder 1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30001 # language-only server python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --language-only \ --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 ``` * EPD Disaggregation ```bash Command theme={null} # encoder 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # encoder 1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --encoder-transfer-backend zmq_to_scheduler \ --port 30001 # prefill 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode prefill \ --language-only \ --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 # decode 0 python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode decode \ --port 30003 # router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://$PREFILL_HOST:30002 \ --decode http://$DECODE_HOST:30003 \ --port 8000 ``` #### gRPC Encoder (EPD) You can run the encoder as a gRPC server while keeping prefill/decode as HTTP. When using gRPC encoders, set `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` for the prefill process so it uses the gRPC receiver. ```bash Command theme={null} # gRPC encoder python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --encoder-only \ --grpc-mode \ --encoder-transfer-backend zmq_to_scheduler \ --port 30000 # prefill (HTTP) - tell it to use gRPC receiver SGLANG_ENCODER_MM_RECEIVER_MODE=grpc \ python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode prefill \ --language-only \ --encoder-urls grpc://127.0.0.1:30000 \ --encoder-transfer-backend zmq_to_scheduler \ --port 30002 # decode (HTTP) python -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-8B-Instruct \ --disaggregation-mode decode \ --port 30003 # router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://$PREFILL_HOST:30002 \ --decode http://$DECODE_HOST:30003 \ --port 8000 ``` # Expert Parallelism Source: https://docs.sglang.io/docs/advanced_features/expert_parallelism Expert Parallelism (EP) in SGLang distributes expert weights across multiple devices in Mixture-of-Experts (MoE) models, addressing memory bottlenecks and enabling efficient scaling for high-performance inference. It is particularly vital for serving large-scale MoE models where tokens are dynamically routed to specialized experts across GPUs. By leveraging optimized all-to-all communication and grouped matrix multiplications (GEMMs), EP reduces latency, boosts throughput, and minimizes idle GPU time. SGLang's EP offers strong extensibility through its modular framework, allowing seamless integration of custom kernels, backends, and optimizations without refactoring core logic, supporting diverse hardware and quantization schemes. ## Supported Backends and Selection Guidance SGLang's EP integrates diverse, highly efficient backends for different use cases, allowing fine-grained control over performance trade-offs. Users specify backends via command-line flags: * `--moe-a2a-backend`: Selects the backend for all-to-all communication. * `--moe-runner-backend`: Selects the backend for MoE computation. ### Backends for All-to-All Communication
Backend Description Use Cases
**`none` (default)** Disables all-to-all for EP. Uses All-Reduce or All-Gather for token dispatch. Hybrid EP and TP setups.
`deepep` DeepEP, a communication library for efficient token shuffling in MoE models. Large-scale EP deployments.
`mooncake` An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers. Elastic EP serving.
nixl NIXL-EP, an elastic EP communication library built on NVIDIA's NIXL framework with native RDMA and NVLink support. Elastic EP serving with fault tolerance and dynamic scaling.
mori MORI-EP, AMD's native all-to-all communication implementation optimized for ROCm. AMD GPU deployments.
`flashinfer` Flashinfer implementation of all-to-all. Large-scale EP deployments.
`ascend_fuseep` Ascend NPU native fused all-to-all communication. Ascend NPU deployments.
`pplx` pplx-kernels, Perplexity's NVSHMEM-based all-to-all dispatch/combine kernels. Low-latency (masked) only; targets FP8 (DeepGEMM) MoE models on Hopper. Requires NVSHMEM 3.2.5, nvshmem4py, cuda-python, and a prebuilt libpplx\_kernels.so (sm\_90a). Low-latency decode EP on Hopper.
DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP and PPLX currently operate in low-latency mode with CUDA Graph support (PPLX reuses the DeepEP low-latency masked expert-compute path). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes. Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep`, `pplx` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. Note that `pplx` additionally requires `--enable-dp-attention` with at least 2 DP groups (i.e., `tp_size / attention_tp_size > 1`); otherwise pplx-kernels' AllToAll cannot be constructed. ### Backends for MoE Computation
Backend Description Use Cases
**`auto` (default)** Automatically selects the optimal backend based on model architecture, hardware (e.g., NVIDIA architecture like Ampere, Hopper, Blackwell), quantization scheme (e.g., FP8, FP4), and runtime conditions. General-purpose deployments; ensures compatibility and performance without user intervention.
`triton` Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create tuned configurations. Custom kernel development or scenarios requiring high extensibility with Torch compilation support.
`deep_gemm` DeepGEMM backend optimized for MoE matrix multiplications, supporting contiguous layouts for prefill and masked layouts for decode; often JIT-compiled for performance. Large-scale EP deployments with FP8 block-wise quantization.
`cutlass` CUTLASS-based backend for efficient GEMMs. NVIDIA architectures with CUTLASS support.
`flashinfer_trtllm` FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs. Blackwell with TRT-LLM.
flashinfer\_trtllm\_routed FlashInfer integrated with TensorRT-LLM for accelerated routed MoE computations, consuming SGLang-computed top-k expert assignments and weights. Compatible with flashinfer all-to-all. Blackwell with TRT-LLM.
`flashinfer_cutlass` FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently. Compatible with flashinfer all-to-all. Blackwell with FP4/FP8 models.
`flashinfer_mxfp4` FlashInfer variant optimized for MXFP4 (mixed FP4) quantization in MoE runners, focusing on memory-efficient low-precision inference. Low-precision models with MXFP4.
`flashinfer_cutedsl` FlashInfer with a custom DSL for flexible and efficient MoE kernel generation, integrated with ModelOpt FP4 quantization. Compatible with flashinfer all-to-all. Low-precision models with NVFP4.
### Examples Launch with DeepEP and DeepGEMM for DeepSeek-V3: ```bash Command theme={null} python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8 ``` ## Extensible EP Framework SGLang's EP framework provides modular abstractions for easy integration of custom kernels, backends, and optimizations. It decouples the MoE forward pass into stages (dispatch → pre-permute → core runner → post-permute → combine), enabling seamless extensions without refactoring core logic. ### Framework Overview The framework centers on `FusedMoE` as the unified entry point for a single, extensible structure. Key components include: * **Dispatcher**: Manages dispatch/combine for backends like DeepEP (implements `BaseDispatcher` subclasses). * **MoeRunner**: Orchestrates grouped-GEMM execution via `MoeRunnerCore` implementations (e.g., `TritonRunnerCore`). * **PermuteMethodPool**: Auto-registers layout conversions (e.g., pre/post-permute via `register_pre_permute` and `register_post_permute` for dynamic modes, or `register_fused_func` for static, torch.compile-compatible fused operations). * **TopK Router**: Backend-agnostic expert selection. This design supports multiple backends via `--moe-a2a-backend` and `--moe-runner-backend`, with quantization integrated through a standardized `apply()` method. The computation flow ensures modularity: ```text Output theme={null} [input_hidden_states] | v TopK.forward -> select_experts / triton_kernels.routing / bypass | v [TopKOutput] | v FusedMoE.forward -> Dispatcher.dispatch -> DeepEP / bypass | | | v | [DispatchOutput] | | | v | quant_method.apply -> MoeRunner.forward | | | | | v | | pre-permute + grouped_gemm + post-permute | | | | |-------------- | v | [CombineInput] | | | v | Dispatcher.combine -> DeepEP / bypass | | |--------------------- v [final_hidden_states] ``` For details, see the [MoE Refactor Roadmap](https://github.com/sgl-project/sglang/issues/8715). ### Implementing New Backends To add a new backend: 1. For a new all-to-all dispatcher, implement a `BaseDispatcher` subclass with `dispatch` and `combine` methods. 2. For a new MoE runner backend, define a `MoeRunnerCore` subclass for core operations (e.g., grouped GEMMs). 3. Define new input/output formats for the dispatcher or model runner (e.g., `RunnerInput`, `RunnerOutput`). 4. Register permute/unpermute methods to ensure compatibility: * **Fused Mode** (static, torch.compile-compatible): Use `register_fused_func` for end-to-end operations. * **Permute Mode** (dynamic): Register `register_pre_permute` and `register_post_permute` for flexible layouts. See the [MoE Refactor Implementation PR](https://github.com/sgl-project/sglang/pull/9269) for full changes, including type hints and config expansions. ### Examples For an example implementation, see [moe\_runner/triton.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/triton.py), which demonstrates Triton-based grouped GEMMs with registered fused and permutation functions. ## Computation and Communication Overlap SGLang's EP employs advanced overlap techniques to hide communication latency behind computation, maximizing GPU utilization in MoE layers. ### Two-Batch Overlap (TBO) TBO splits requests into micro-batches, interleaving attention computation with dispatch/combine operations. Yield points in the execution graph allow pausing for overlaps, increasing overall throughput without peak memory spikes: ```python Example theme={null} operations = [ self._forward_attn, YieldOperation(), # Overlap with dispatch of prior micro-batch self._forward_dispatch, self._forward_mlp, YieldOperation(), # Overlap with combine self._forward_combine, ] ``` Users need to specify `--enable-two-batch-overlap` to unlock up to 2x throughput. For details, see the [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#two-batch-overlap). ### Single-Batch Overlap (SBO) SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabling the overlap of operations within a single batch—such as shared experts computation with communication—while decentralizing logic to enhance modularity. These hooks execute before and after the `dispatch` and `combine` operations without modifying core MoE modules. This design simplifies interfaces, reduces coupling, and improves extensibility. For implementation details and an example of overlapping shared experts with DeepEP's combine operation, refer to [PR #13327](https://github.com/sgl-project/sglang/pull/13327). Users can set `--enable-single-batch-overlap` to enable this feature. ## Workload Balancer SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability. To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains. For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb). ## Ascend NPU Guidance ### Guidance on SGLang configuration in Ascend NPU * `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends, * `deepep`: The mechanism is consistent with the above description. * `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode. * `--moe-runner-backend` parameter does not need to be configured. * `--deepep-mode`: * In PD mixed mode, please set `--deepep-mode auto`. * In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`. ### DeepEP Ascend Introduction DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP). It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences. Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables: * `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192. * `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1. * `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled). `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase. The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows: ```text Output theme={null} # Enable Ant-moving Function HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE # Disable Ant-moving Function HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE ``` Wherein the parameters are described as follows: * `hidden_size`: hidden size in model config. * `topk`: The number of selected routing experts. * `TOTAL_SEQ_LEN`: input sequence length. * `PADDING_BUFFSIZE`: A value of 20 or greater is recommended. # Hierarchical KV Caching (HiCache) Source: https://docs.sglang.io/docs/advanced_features/hicache * [Hicache Best Practices](./hicache_best_practices) * [Hicache Design](./hicache_design) * [Hicache Storage Runtime Attach Detach](./hicache_storage_runtime_attach_detach) # SGLang HiCache Best Practices Source: https://docs.sglang.io/docs/advanced_features/hicache_best_practices ## Why HiCache Matters SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems. ## Configuration Guidelines ## Core HiCache Parameters ```bash Command theme={null} # Essential HiCache flags --page-size 64 # Page size for cache management --enable-hierarchical-cache # Enable HiCache --hicache-ratio 2 # Host memory ratio (2x GPU memory) --hicache-size 100 # Host memory size in GBs, will override the above ratio --hicache-io-backend kernel # The I/O backend of moving data between CPU and GPU --hicache-write-policy write_through # Cache write policy from GPU to CPU --hicache-storage-backend # Optional storage backend (e.g., hf3fs, mooncake, etc.) ``` Notes: * Besides configuring `--hicache-storage-backend` at startup, SGLang also supports **runtime attach/detach** of the HiCache storage backend (no restart required) via HTTP admin endpoints. See [Runtime Attach/Detach HiCache Storage Backend](./hicache_storage_runtime_attach_detach). ## Key Configurations with Storage Backends Enabled ### Memory Layout Optimization ```bash Command theme={null} # Page-first: Optimized for I/O efficiency with zero-copy (recommended with kernel backend) --hicache-mem-layout page_first # Page-first-direct: Optimized for direct I/O operations (Compatible with fa3 and same zero-copy performance as page_first) --hicache-mem-layout page_first_direct # Layer-first --hicache-mem-layout layer_first ``` **Layout Compatibility:** * `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend * `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization ### Heterogeneous TP Support (GQA/MHA models) HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace. Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`: ```bash Command theme={null} # Example: heterogeneous TP = {4, 8}, so lcm = 8 --hicache-storage-backend-extra-config '{"tp_lcm_size": 8}' ``` Guidelines: * Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage. * For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments. * If all clusters use the same TP size, this option is not needed. ### Prefetch Policies ```bash Command theme={null} # Best-effort: Terminate prefetch when needed --hicache-storage-prefetch-policy best_effort # Wait-complete: Ensure complete prefetch, higher cache reuse --hicache-storage-prefetch-policy wait_complete # Timeout: Balance between completion and best-effort --hicache-storage-prefetch-policy timeout ``` ### Integration with PD Disaggregation HiCache works seamlessly with PD Disaggregation. You can choose between two configurations: 1. **Prefill-only HiCache**: Enable HiCache only on Prefill nodes, allowing KV cache sharing among Prefill instances 2. **Full HiCache with async offloading**: Enable HiCache on Prefill nodes and async KV cache offloading on Decode nodes, allowing Prefill nodes to reuse KV caches from Decode nodes in multi-turn dialogue scenarios ```bash Command theme={null} # Prefill node with HiCache enabled for cross-prefill sharing (ideal for SystemPrompt scenarios) python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --mem-fraction-static 0.85 \ --page-size 64 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ --disaggregation-ib-device mlx5_0 \ --disaggregation-mode prefill \ --disaggregation-transfer-backend mooncake # Decode node with async offloading enabled for KV cache reuse by Prefill (ideal for multi-turn conversations) python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --page-size 64 \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ --disaggregation-decode-enable-offload-kvcache \ # Enable async KV cache offloading in decode node --disaggregation-ib-device mlx5_0 \ --disaggregation-mode decode \ --disaggregation-transfer-backend mooncake ``` ### Deployment with HF3FS Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/hf3fs/docs/README.md). ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /xxx/DeepSeek-R1/ \ --log-level info \ --tp 8 \ --host 0.0.0.0 \ --port 10000 \ --enable-metrics \ --enable-cache-report \ --page-size 64 \ --mem-fraction-static 0.85 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-size 0 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-write-policy write_through \ --hicache-storage-backend hf3fs \ --hicache-storage-prefetch-policy wait_complete \ ``` ### Deployment with Mooncake Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md). ```bash Command theme={null} # Set Mooncake environment variables export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata" export MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240 export MOONCAKE_PROTOCOL="rdma" export MOONCAKE_DEVICE="$DEVICE_LIST" export MOONCAKE_MASTER=127.0.0.1:50051 # Launch SGLang server with Mooncake backend python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --tp 8 \ --page-size 64 \ --enable-hierarchical-cache \ --hicache-ratio 2 \ --hicache-mem-layout page_first_direct \ --hicache-io-backend direct \ --hicache-storage-backend mooncake \ --hicache-write-policy write_through \ --hicache-storage-prefetch-policy timeout ``` ## Custom Storage Backend Integration To integrate a new storage backend: 1. **Implement three core methods:** * `get(key)`: Retrieve value by key * `exists(key)`: Check key existence * `set(key, value)`: Store key-value pair 2. **Register your backend:** Add your storage backend to the HiCache [BackendFactory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/backend_factory.py#L188) The HiCache controller handles all scheduling and synchronization automatically. ### Dynamic Backend Loading Alternatively, you can use dynamic loading to avoid hard-coding your backend in the repository: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path your-model \ --enable-hierarchical-cache \ --hicache-storage-backend dynamic \ --hicache-storage-backend-extra-config '{"backend_name":"custom_backend_name", "module_path": "your_module_path", "class_name": "YourHiCacheClassName"}' ``` **Configuration Parameters:** * `--hicache-storage-backend`: Set to `dynamic` * `--hicache-storage-backend-extra-config`: JSON configuration with: * `backend_name`: Custom backend identifier * `module_path`: Python module path to your implementation * `class_name`: Your HiCache implementation class name * `interface_v1`: 0 (disable) or 1 (enable) to control usage of batch\_get\_v1 and batch\_set\_v1 methods ## Community and Support * **GitHub Issues**: Report bugs and feature requests * **Slack Channel**: Join community discussions in #sgl-kv-cache-store * **Documentation**: Refer to storage backend-specific guides *** *This document will be continuously updated based on community feedback and new features. Contributions and suggestions are welcome!* # HiCache System Design and Optimization Source: https://docs.sglang.io/docs/advanced_features/hicache_design This document provides a comprehensive overview of SGLang HiCache, covering its system architecture, workflow and key components. It also details configuration parameters, optimization techniques, and integration with various L3 storage backends, serving as a complete reference for users and developers to understand and tune HiCache for efficient LLM inference. ## Why and What is HiCache? In large language model inference, the prefill phase is often time-consuming: input sequences need to be first converted into Key-Value cache (KV cache) for subsequent decoding. When multiple requests share the same prefix, the KV cache for that prefix is identical. By caching and reusing these shared KV caches, redundant computation can be avoided. To address this, SGLang introduced RadixAttention, which leverages idle GPU memory to cache and reuse prefix KV caches, and **HiCache**, which extends this idea to host memory and distributed storage. Inspired by the classic three-level cache design of modern CPUs, HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3. This hierarchy enables HiCache to fully exploit the "idle" storage space of GPUs and CPUs, while integrating distributed cache systems such as Mooncake, 3FS, NIXL, and AIBrix KVCache for global KV cache storage and scheduling. As a result, HiCache significantly expands KV cache capacity while maintaining strong read performance—especially in workloads such as multi-QA and long-context inference, where KV cache reuse is frequent. For detailed benchmark results, see [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/). ## System Design ### Overall Architecture In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster. ### HiRadixTree: Metadata Organization in HiCache For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage. HiRadixTree extends this idea: each node corresponds to the KV cache of a span of consecutive tokens and records where that KV cache is stored—whether in local GPU memory, CPU memory, L3 storage, or multiple of these tiers. If stored locally, HiRadixTree maintains precise metadata, including the exact storage address. However, to reduce overhead, HiRadixTree does not store or continuously synchronize metadata for L3 KV cache. Instead, when accessing L3 data, it queries the backend in real time to retrieve the necessary metadata, such as whether the data exists and on which server and location it resides. ### Overall Workflow The workflow of HiCache mainly involves three key operations: **local match**, **prefetch** and **write-back**. When the system receives a new request, it first searches the local L1 and L2 caches for matching KV caches. For parts not found locally, it attempts to prefetch from L3. After prefetching, all required KV caches are loaded into the GPU for computation. Once the prefill computation is complete, the system considers storing the newly generated data into L2 or L3. HiCache Workflow ### Local Match Local matching is the first step in HiCache's workflow, where incoming request tokens are matched against the HiRadixTree to locate cached KV data in local memory tiers (L1 GPU memory and L2 host memory). The matching algorithm traverses the HiRadixTree from the root node, following child nodes that match the token sequence prefix. At each node, the incoming token sequence is compared with the node’s stored token sequence. When `page_size > 1`, matching is performed at the page granularity to optimize memory access patterns. If a match terminates within a node’s stored sequence, the node is automatically split to create an exact boundary, improving the efficiency of future matches. The algorithm returns a continuous prefix of the request, with the first part residing in L1 and the latter part in L2. Since the process only requires traversing the local HiRadixTree and does not involve any actual data copying, local matching is extremely fast. ### Prefetch from L3 Data prefetching is one of HiCache’s core optimization techniques, designed to proactively load KV caches from L3 storage into local L2 memory, thereby reducing access latency during subsequent operations. **Prefetch Trigger Conditions**: After local matching, for the parts not found in L1 or L2, the system queries L3 to retrieve metadata for the next continuous matching KV caches. If the length of hit cache in L3 exceeds a threshold (default: 256 tokens, configurable), a prefetch operation is triggered. **Prefetch Strategies**: HiCache provides three different prefetch termination strategies to address different scenario needs: * **best\_effort**: Terminates immediately when GPU can execute prefill computation, with no waiting time, suitable for scenarios extremely sensitive to latency. * **wait\_complete**: Must wait for all prefetch operations to complete, suitable for scenarios requiring high cache hit rates. * **timeout**: Terminates after specified time or when complete, balancing latency and cache hit rate needs. After prefetching stops, the data already fetched is used together with the local data for the prefill computation. For **timeout** strategy, HiCache introduces three configuration parameters to support fine-grained control over prefetch timeout conditions: * `prefetch_timeout_base`: the base timeout, representing overhead unrelated to the number of tokens (e.g., scheduling and synchronization). Default: `2` seconds. * `prefetch_timeout_per_ki_token`: the incremental timeout per thousand tokens. Default: `0.1` seconds per 1024 tokens. * `prefetch_timeout_max`: the upper bound applied to the linear timeout, preventing very long prompts from waiting unboundedly. Default: `30` seconds. The timeout is computed as: ```python Example theme={null} timeout = min( prefetch_timeout_max, prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024, ) ``` ### Data Write-back The write-back mechanism is responsible for moving frequently accessed KV caches from L1 to L2 and L3, enabling larger and longer-term storage as well as cache sharing across instances. **Configurable Write-back Policies**: HiCache supports three write-back strategies: * **write\_through**: Every access is immediately written back to the next level. When bandwidth is sufficient, this strategy provides the strongest caching benefit. * **write\_through\_selective**: Data is written back only after the access frequency exceeds a threshold. This strategy backs up only hot data, reducing I/O overhead. * **write\_back**: Data is written back to the next level only when it is evicted from the upper level. This strategy alleviates storage pressure and is suitable for scenarios where storage capacity is limited but memory utilization must be maximized. **Cross-instance Sharing**: When data is written back from L2 to L3, only data not already present in L3 is transferred. KV caches stored in L3 can then be shared across all SGLang instances in the cluster (depending on the L3 backend implementation), significantly improving cache hit rates within the same memory budget. ### Multi-Rank Synchronization During multi-GPU parallel computation, such as tensor parallelism (TP), HiCache must ensure consistent states across different ranks. Therefore, critical computation steps require the use of `all_reduce` for state synchronization. For example, during prefetching, `all_reduce(op=min)` is used to ensure that all ranks obtain the same number of L3 hits, preventing inconsistent judgments about whether the prefetch threshold has been reached. Similarly, after prefetching completes or terminates, `all_reduce(op=min)` is again required to guarantee consensus among ranks on the prefix length of the successfully retrieved KV cache. ### Data Transfer Optimization **Zero-Copy Data Transfers**: Both prefetching and write-back involve substantial data movement. Minimizing the number of data copies can significantly improve system performance. HiCache supports passing memory addresses and sizes directly when transferring data from L2 memory to an L3 backend. **“Batch-Oriented” Data Organization**: The granularity of data reads and writes has a major impact on performance. To address this, HiCache L3 stores and transfers KV cache data at the granularity of **pages** and supports different data layouts beyond the existing `layer first` scheme, including `page first` and `page first direct`. Under the `page first` and `page first direct` layouts, all KV cache data belonging to the same page is placed in contiguous memory, allowing it to be passed as a single object to L3 using zero-copy transfers. HiCache L2 MEM layout However, because GPU KV computation is naturally performed layer by layer, the GPU inherently operates in a `layer first` layout. When transferring `page first` data from L2 to the GPU, data must be transferred at the granularity of one token per layer. The `page first direct` layout mitigates this issue by grouping together all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level. **CPU-to-GPU Transfer Optimizations**: In HiCache, moving data from CPU memory to GPU is as performance-critical as prefetching data from L3 to L2. HiCache employs several optimizations for this process: * **Compute-Transfer Overlap**: During the prefill phase, when transferring data from CPU to GPU, HiCache overlaps layers by concurrently loading the KV cache of layer N+1 while computing layer N. This effectively hides data transfer latency. * **GPU-assisted I/O Kernels**: On top of `cudaMemcpyAsync`, HiCache implements a set of GPU-assisted I/O kernels specifically optimized for KV cache transfers between CPU and GPU. Compared to the baseline approach, these kernels achieve up to 3x higher transfer speed. **Write-back Optimization for MLA**: For MHA (Multi-Head Attention) models under multi-TP, each rank holds `1/tp_size` of a token’s KV data. In contrast, for MLA (Multi-Layer Attention) models, all ranks hold the complete and identical KV data for each token. HiCache includes a dedicated optimization for MLA: only one rank initiates the write-back operation, ensuring that data is not redundantly stored across ranks. ### Integration with PD-Disaggregation Deployment Mode SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](./pd_disaggregation)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3. ### Unified Interfaces and Rich L3 Storage Backends HiCache encapsulates all read, write, and query operations on L3 backends within the `class HiCacheStorage(ABC)`, exposing a set of simple and consistent interfaces. This design supports a wide range of L3 storage backends and allows users to select the one that best fits their specific use cases. * **Mooncake**: Mooncake is a high-performance caching system for LLM inference that leverages RDMA and multi-NIC resources to enable zero-copy, ultra-fast data transfers. Try Mooncake [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store). * **DeepSeek 3FS (HF3FS)**: HF3FS is a Kubernetes-native distributed storage solution with operator-based deployment. Try HF3FS [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs). * **NIXL**: NIXL provides a unified API for accessing various storage plugins, including but not limited to DeepSeek's 3FS, GPU Direct Storage (GDS) and Amazon S3-compatible object storage. Try NIXL [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/nixl). * **AIBrix KVCache**: AIBrix KVCache is a production-ready KVCache Offloading Framework, which enables efficient memory tiering and low-overhead cross-engine reuse. Try AIBrix KVCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/aibrix_kvcache). * **HiCacheFile**: A simple file-based storage backend for demonstration purposes. Specifically, **LMCache**, an efficient KV cache layer for enterprise-scale LLM inference, provides an alternative solution to HiCache. Try LMCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/lmcache). ## Related Parameters * **`--enable-hierarchical-cache`**: Enable hierarchical cache functionality. This is required to use HiCache. * **`--hicache-ratio HICACHE_RATIO`**: The ratio of the size of host KV cache memory pool to the size of device pool. For example, a value of 2 means the host memory pool is twice as large as the device memory pool. The value of this parameter must be greater than 1, as the current implementation requires the host memory allocated for the KV cache to be larger than the device memory allocated for the KV cache. * **`--hicache-size HICACHE_SIZE`**: The size of host KV cache memory pool in gigabytes. This parameter overrides `hicache-ratio` if set. For example, `--hicache-size 30` allocates 30GB (1GB = 1e9 bytes) for the host memory pool **for each rank**. If there are 8 ranks, then the total memory size is 240GB. Just like `hicache-ratio`, the value of this parameter must be larger than the size of device memory allocated for KV cache. **Note**: `--hicache-ratio` and `--hicache-size` are two critical parameters. In general, a larger HiCache size leads to a higher cache hit rate, which improves prefill performance. However, the relationship between cache size and hit rate is not linear. Once most reusable KV data—especially hot tokens—are already cached, further increasing the size may yield only marginal performance gains. Users can set these parameters based on their workload characteristics and performance requirements. * **`--page-size PAGE_SIZE`**: The number of tokens per page. This parameter determines the granularity of KV cache storage and retrieval. Larger page sizes reduce metadata overhead and improve I/O efficiency for storage backends, but may lower the cache hit rate when only part of a page matches the stored KV cache. For workloads with long common prefixes, larger pages can improve performance, while workloads with more diverse prefixes may benefit from smaller pages. See [Data Transfer Optimization](#data-transfer-optimization) for how page granularity affects I/O performance. * **`--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`**: Controls when prefetching from storage should stop. See [Prefetch from L3](#prefetch-from-l3) for details. * `best_effort`: Prefetch as much as possible without blocking * `wait_complete`: Wait for prefetch to complete before proceeding * `timeout`: Terminates after specified time or when complete (Recommended for production environments, as setting an appropriate timeout helps the system meet required SLOs) * **`--hicache-write-policy {write_back,write_through,write_through_selective}`**: Controls how data is written from faster to slower memory tiers. See [Data Write-back](#data-write-back) for details. * `write_through`: Immediately writes data to all tiers (strongest caching benefits) * `write_through_selective`: Uses hit-count tracking to back up only frequently accessed data * `write_back`: Writes data back to slower tiers only when eviction is needed (reduces I/O load) * **`--hicache-io-backend {direct,kernel}`**: Choose the I/O backend for KV cache transfer between CPU and GPU. See [Data Transfer Optimization](#data-transfer-optimization) for details. * `direct`: Standard CUDA memory copy operations * `kernel`: GPU-assisted I/O kernels (recommended for better performance) * **`--hicache-mem-layout {layer_first,page_first,page_first_direct}`**: Memory layout for the host memory pool. See [Data Transfer Optimization](#data-transfer-optimization) for details. * `layer_first`: Compatible with GPU computation kernels (default for GPU memory) * `page_first`: Optimized for I/O efficiency * `page_first_direct`: Groups all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level * **`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`**: Choose the storage backend for the L3 tier. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: `backend_name` (custom name), `module_path` (Python module path), `class_name` (backend class name). See [Unified Interfaces and Rich L3 Storage Backends](#unified-interfaces-and-rich-l3-storage-backends) for available backends. * **`--enable-lmcache`**: Using LMCache as an alternative hierarchical cache solution. * **`--lmcache-config-file`**: Path to the LMCache YAML configuration file. * **`--hicache-storage-backend-extra-config HICACHE_STORAGE_BACKEND_EXTRA_CONFIG`**: the extra config can be either * a JSON string containing extra configuration for the storage backend, e.g., `--hicache-storage-backend-extra-config '{"prefetch_threshold":512, "prefetch_timeout_base": 0.5, "prefetch_timeout_per_ki_token": 0.25}' `, or * a TOML or JSON or YAML file specifying the extra configuration for the storage backend (to differentiate from the JSON string input, prepend a `@` in front of the file name), e.g., `--hicache-storage-backend-extra-config "@config.toml"` where `config.toml` is the config file containing the complex configurations. This can be useful when the configuration consists of many or complex key-value pairs (for instance, it is preferred to use a config file for NIXL backend as its configurations can be complex). # Runtime Attach/Detach HiCache Storage Backend (No Restart) Source: https://docs.sglang.io/docs/advanced_features/hicache_storage_runtime_attach_detach This document explains how to **dynamically attach/detach the HiCache L3 storage backend at runtime** (e.g., `mooncake` / `hf3fs` / `nixl` / `file` / `aibrix` / `eic`) while **SGLang is already running and serving traffic**, without restarting the process. For safety and consistency, the current implementation **strictly requires** these operations to happen only when the service is **idle**: * **No running requests** * **No waiting/queued requests** If the idle condition is not met, the API will fail fast (HTTP 400) and **will not modify** the current service state. *** ## 1. Background and implementation overview ### 1.1 Architecture / control path The control path is: 1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`) * Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend` 2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_control_mixin.py`) * Sends the request to the Scheduler via `FanOutCommunicator` 3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`) * Performs a **strict idle check** * Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)` 4. **HiRadixCache** (`python/sglang/srt/mem_cache/hiradix_cache.py`) * Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs) * Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)` 5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`) * Creates/destroys the storage backend instance (via `StorageBackendFactory`) * Starts/stops backend background threads at runtime (prefetch/backup) *** ## 2. Idle-state requirement (strict) The Scheduler uses `is_fully_idle()` which checks: * No running batches (including chunked prefill, overlap, pipeline-parallel, and disaggregation paths) * No waiting requests in any queue (waiting, grammar, disagg bootstrap/prealloc/transfer/inflight) * No DLLM staging requests If the condition is not met, attach/detach returns an error like: * `Reject attach: scheduler is not idle. #queue-req=... #running-req=...` before switching, drain upstream traffic and wait for the server to become idle, then call attach/detach. ### 2.1 DP (data parallel) semantics When `dp_size > 1`, the tokenizer dispatches the request to **all DP scheduler instances** and aggregates their responses: * The final `success` is **true only if all DP ranks return success** * The final `message` concatenates messages from all DP ranks This is intended to prevent “silent partial success”, but it also means you may see: * Overall **failure** even though **some ranks already succeeded** Currently there is **no automatic partial rollback** across DP ranks (see TODO in code). Operationally: * Prefer to keep backend config identical across ranks * If attach fails, immediately call detach (best-effort/idempotent), fix config, then retry attach *** ## 3. How to use (HTTP Admin API) The examples below assume your SGLang HTTP server is at `http://127.0.0.1:30000`. ### 3.1 Query current storage backend status ```bash Command theme={null} curl -s http://127.0.0.1:30000/hicache/storage-backend ``` Example response: ```json Config theme={null} { "hicache_storage_backend": "mooncake", "hicache_storage_backend_extra_config": "{\"master_server_address\":\"127.0.0.1:50051\", ...}" } ``` ### 3.2 Attach (enable) a storage backend ```bash Command theme={null} curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \ -H 'Content-Type: application/json' \ -d '{ "hicache_storage_backend": "mooncake" }' ``` ```bash Command theme={null} curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \ -H 'Content-Type: application/json' \ -d '{ "hicache_storage_backend": "mooncake", "hicache_storage_backend_extra_config_json": "{\"master_server_address\":\"127.0.0.1:50051\",\"protocol\":\"tcp\",\"global_segment_size\":\"4gb\",\"prefetch_threshold\":256}", "hicache_storage_prefetch_policy": "timeout" }' ``` Notes: * `hicache_storage_backend_extra_config_json` can include both: * **Backend configuration** (e.g., Mooncake master/metadata/protocol, etc.) * **Prefetch configuration** (`prefetch_threshold`, `prefetch_timeout_base`, `prefetch_timeout_per_ki_token`, `prefetch_timeout_max`, `hicache_storage_pass_prefix_keys`) ### 3.3 Detach (disable) the storage backend ```bash Command theme={null} curl -s -X DELETE http://127.0.0.1:30000/hicache/storage-backend ``` Notes: * Detach only makes SGLang **stop using** the L3 storage backend and stops prefetch/backup threads * It **does not automatically delete** data stored in Mooncake/HF3FS (or other remote backends) *** ## 4. Behavior and caveats * **No restart required**: attach/detach switches in-process at runtime * **Must be idle**: otherwise the request is rejected to avoid consistency issues * **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error * **Observability**: * After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides * If metrics are enabled, attach will create a storage metrics collector in `HiRadixCache` on demand # HiSparse: Hierarchical Sparse Attention Source: https://docs.sglang.io/docs/advanced_features/hisparse_guide HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency. > **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. ## Why HiSparse? In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by: * **Reducing GPU memory per request**: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length. * **On-demand swap-in**: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores. * **Transparent to prefill**: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes. ## Design Overview ### Decode Workflow Each decode step follows this flow: 1. **Forward decode** — generate the next token 2. **Top-k selection** — select the most relevant token positions via attention scores 3. **Swap-in** — the CUDA kernel loads top-k KV entries from host to device buffer: * *Short sequences* (`seq_len ≤ device_buffer_size`): fast path, all KV already in buffer * *Long sequences*: hit detection → LRU reordering → miss handling (host → device copy) 4. **Decode attention** — compute attention using the top-k device locations 5. **Eager backup** — asynchronously copy the previous token's KV from device to host ### PD Disaggregation Integration (Direct-to-Host) In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step. ``` Prefill GPU ──RDMA──▶ Decode Host Pool (CPU pinned memory) │ ▼ alloc device buffer (4KB) │ ▼ swap-in kernel (on-demand top-k) ``` For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4\_indexer and C128 KV remain device-to-device transfers. ## Server Arguments
Argument Type / Default Description
--enable-hisparse flag; default: disabled Enable HiSparse on the decode instance
--hisparse-config JSON string Configuration for HiSparse (see below)
### HiSparse Config Parameters Pass as a JSON string via `--hisparse-config`:
Parameter Type / Default Description
top\_k int Number of topk entries
device\_buffer\_size int Number of token slots in the per-request GPU device buffer
host\_to\_device\_ratio int Ratio of logical pool size to device pool size, determining host memory capacity
swap\_in\_block\_size int / 960 CUDA thread-block size for the HiSparse swap-in kernel
Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'` ### Shared-index prefetch (automatic) When a model reuses one anchor layer's top-k selection across a run of subsequent "skip" layers (DSA `index_topk_freq` / `index_topk_pattern`; native in GLM-5.2 as IndexShare), the working set of every skip layer is known the moment the anchor's index is computed. HiSparse exploits this automatically: the anchor's swap-in kernel records its miss plan (which host slots go to which device-buffer slots), and each skip layer replays that plan with a copy-only kernel issued ahead on a side stream, so the skip layers' host→device IO overlaps the intervening layers' compute instead of sitting on the decode critical path. The replay kernel uses a small fixed grid to keep its SM footprint low while overlapped. The prefetch is enabled automatically for eligible models (no pipeline parallelism, no speculative decoding) and can be turned off for A/B comparison with `SGLANG_DISABLE_HISPARSE_PREFETCH=1`. ## Deployment HiSparse currently requires **PD disaggregation mode** and is enabled only on the **decode instance**. ### Prefill Instance ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/model \ --trust-remote-code \ --port 8000 --host 0.0.0.0 \ --context-length 81920 \ --chunked-prefill-size 65536 \ --tp-size 8 --dp-size 8 --enable-dp-attention \ --mem-fraction-static 0.85 \ --disaggregation-mode prefill \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --nnodes 1 --node-rank 0 ``` ### Decode Instance (with HiSparse) ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/model \ --trust-remote-code \ --port 8000 --host 0.0.0.0 \ --context-length 81920 \ --tp-size 8 --dp-size 8 --enable-dp-attention \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --disaggregation-mode decode \ --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \ --dist-init-addr 127.0.0.1:5757 \ --nnodes 1 --node-rank 0 \ --enable-hisparse \ --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}' ``` > **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`), except for GLM DSA models on SM120/SM121 with `fp8_e4m3`, which use `flashinfer_sparse_mla`. DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend. ### Benchmark ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ --dataset-name random \ --random-input 40000 \ --random-output 20000 \ --num-prompts 200 \ --max-concurrency 200 \ --request-rate 40 \ --random-range-ratio 1.0 \ --host 127.0.0.1 \ --port 20000 \ --model /path/to/model \ --flush-cache \ ``` ### Key Notes * The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse. * On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse. * For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`. * On SM120/SM121 (e.g. RTX PRO 6000, RTX 5090) with GLM DSA models and `--kv-cache-dtype fp8_e4m3`, both DSA backends resolve to `flashinfer_sparse_mla`, which is the only DSA kernel available on that architecture. HiSparse accepts it there; no extra flag is needed. * For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default. * `host_to_device_ratio` should be configured based on the host machine's available memory. For example: * **\~1 TB** host memory → `host_to_device_ratio: 5` * **\~2 TB** host memory → `host_to_device_ratio: 10` ## Acknowledgments We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions. # Hyperparameter Tuning Source: https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning ## Achieving high throughput for offline batch inference Achieving a large batch size is the most important thing for attaining high throughput in offline batch inference. When the server is running at full load in a steady state, look for the following in the log: ```text Output theme={null} Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317 ``` ### Adjust the request submission speed to control `#queue-req` `#queue-req` indicates the number of requests in the queue. If you frequently see `#queue-req: 0`, it suggests that your client code is submitting requests too slowly. A healthy range for `#queue-req` is `100 - 2000`. However, avoid making `#queue-req` too large, as this will increase the scheduling overhead on the server. ### Achieve a high `token usage` `token usage` indicates the KV cache memory utilization of the server. `token usage > 0.9` means good utilization. If you frequently see `token usage < 0.9` and `#queue-req > 0`, it means the server is too conservative about taking in new requests. You can decrease `--schedule-conservativeness` to a value like 0.3. The case of a server being too conservative can happen when users send many requests with a large `max_new_tokens` but the requests stop very early due to EOS or stop strings. On the other hand, if you see `token usage` very high and you frequently see warnings like `KV cache pool is full. Retract requests. #retracted_reqs: 1, #new_token_ratio: 0.9998 -> 1.0000`, you can increase `--schedule-conservativeness` to a value like 1.3. If you see `KV cache pool is full. Retract requests.` occasionally but not frequently (\~1 time per minute), it is okay. ### Tune `--mem-fraction-static` to increase KV cache pool capacity SGLang allocates memory as follows: Total memory usage = model weights + KV cache pool + CUDA graph buffers + activations The `--mem-fraction-static` parameter determines how much memory is allocated to the first two components: mem\_fraction\_static = (model weights + KV cache pool) / GPU memory capacity To support higher concurrency, you should maximize the KV cache pool capacity by setting `--mem-fraction-static` as high as possible while still reserving enough memory for activations and CUDA graph buffers. SGLang uses simple heuristics to set the default value of `--mem-fraction-static`, but you can optimize it for your use cases. As a rule of thumb, reserving 5–8 GB of memory for activations is typically sufficient. You can check this by inspecting the logs just before the server is ready. Look for log entries like this: ```text Output theme={null} [2025-08-11 17:17:03] max_total_num_tokens=665690, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=65536, available_gpu_mem=13.50 GB ``` Check the `available_gpu_mem` value. * If it is between 5–8 GB, the setting is good. * If it is too high (e.g., 10 - 20 GB), increase `--mem-fraction-static` to allocate more memory to the KV cache. * If it is too low, you risk out-of-memory (OOM) errors later, so decrease `--mem-fraction-static`. Another straightforward approach is to increase `--mem-fraction-static` in increments of 0.01 until you encounter OOM errors for your workloads. ### Avoid out-of-memory errors by tuning `--chunked-prefill-size`, `--mem-fraction-static`, and `--max-running-requests` If you encounter out-of-memory (OOM) errors, you can adjust the following parameters: * If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts. * If OOM occurs during decoding, try lowering `--max-running-requests`. * You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput. ### Tune `--cuda-graph-max-bs-decode` By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256). However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768. Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value. Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time. ### Tune `--dp-size` and `--tp-size` Data parallelism is better for throughput. When there is enough GPU memory, always favor data parallelism for throughput. Refer to [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for a better data parallelism rather than using `dp_size` parameter. ### Try other options * `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`. * Try other quantization (e.g. FP8 quantization with `--quantization fp8`) * Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--enable-dp-attention --dp-size 8`). * If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead. # llm-d Source: https://docs.sglang.io/docs/advanced_features/llm-d [llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference framework for serving large language models at scale across a fleet of inference servers. SGLang is a supported inference engine in llm-d: llm-d coordinates a fleet of SGLang instances across a cluster so that performance holds up under real production traffic, achieving the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators. llm-d is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/) founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. ## What llm-d adds to an SGLang deployment A single SGLang server is fast, and RadixAttention already maximizes cache reuse within each replica. But at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing as related requests scatter and radix-cache hit rates collapse, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that the engine does not aim to provide on its own: * **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, the llm-d Router scores each replica on prefix-cache locality and current load, routing each request to the replica most likely to already hold its prefix — raising RadixAttention hit rates on multi-turn and shared-prefix workloads while avoiding saturated servers. * **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and tiered offloading spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM. * **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate workers, with KV-cache moved over high-speed interconnects, lowering TTFT and steadying per-token latency on long prompts. * **[SLO-aware autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale SGLang pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch. * **One control plane for mixed fleets.** llm-d schedules across engines, so platform teams can serve SGLang and vLLM pools behind the same gateway, policies, and observability instead of running parallel stacks. These capabilities are composable. Most teams start by adding prefix-aware routing over an existing SGLang pool, then layer in the rest as specific bottlenecks appear. ## Kubernetes-native gateway llm-d builds on the Gateway API Inference Extension, so SGLang pools are managed through standard Kubernetes resources (Gateway, HTTPRoute, InferencePool) and work with supported gateway providers such as Istio, GKE Inference Gateway, and agentgateway, rather than a bespoke routing tier. ## Performance llm-d publishes reproducible benchmarks from production-scale deployments on [Prism](https://prism.llm-d.ai/). One representative result: prefix-aware routing delivered **3x higher output throughput and 2x faster TTFT** than round-robin load balancing (Llama 3.1 70B). The mechanism carries over directly to SGLang, where RadixAttention makes the cluster-level cache hit rate a function of routing. ## Get started * **Deploy the optimized baseline with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart), selecting SGLang as the inference server.** It stands up an intelligent router (the llm-d Router) over an SGLang pool on Kubernetes in a tested configuration. * Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload. * Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the scheduler, gateway, and model servers wrap your SGLang deployment. Questions and contributions are welcome on [GitHub](https://github.com/llm-d/llm-d) and [Slack](https://llm-d.ai/slack). ## Current scope SGLang is supported across the well-lit paths — including intelligent inference scheduling, precise prefix-cache routing (SGLang publishes KV-cache events that the llm-d Router subscribes to), tiered KV-cache management, prefill/decode disaggregation, flow control, and autoscaling. The one current exception is Multi-Node Wide Expert Parallelism, which is vLLM-specific today. See the llm-d documentation for the latest per-engine support status. # LoRA Serving Source: https://docs.sglang.io/docs/advanced_features/lora SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs. ## Arguments for LoRA Serving The following server arguments are relevant for multi-LoRA serving: * `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility. * `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. * `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: \ | \=\ | JSON with schema \{"lora\_name":str,"lora\_path":str,"pinned":bool}. * `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8. * `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`. * `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out. * `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added. * `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. * `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters. * `max_lora_chunk_size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16. * `lora_drain_wait_threshold`: When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default). * `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper. From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to. ## Usage ### Serving Single Adaptor **Note:** SGLang supports LoRA adapters through two APIs: 1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions#using-lora-adapters) for examples. 2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below). ```python Example theme={null} import json import requests from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, terminate_process ``` ```python Example theme={null} server_process, port = launch_server_cmd( # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ --max-loras-per-batch 2 \ --log-level warning \ """ ) wait_for_server(f"http://localhost:{port}") ``` ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses the base model "lora_path": ["lora0", None], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output 0: {response.json()[0]['text']}") print(f"Output 1: {response.json()[1]['text']}") ``` ```python Example theme={null} terminate_process(server_process) ``` ### Serving Multiple Adaptors ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \ --max-loras-per-batch 2 \ --log-level warning \ """ ) wait_for_server(f"http://localhost:{port}") ``` ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output 0: {response.json()[0]['text']}") print(f"Output 1: {response.json()[1]['text']}") ``` ```python Example theme={null} terminate_process(server_process) ``` ### Dynamic LoRA loading Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API. When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly "smaller". ```python Example theme={null} lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json" # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora" # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora" # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj # The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified. # We are adding it here just to demonstrate usage. server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --cuda-graph-max-bs-decode 2 \ --max-loras-per-batch 2 \ --max-lora-rank 256 --lora-target-modules all --log-level warning """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` Load adapter lora0 ```python Example theme={null} response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora0", "lora_path": lora0, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Load adapter lora1: ```python Example theme={null} response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora1", "lora_path": lora1, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Check inference output: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0: \n{response.json()[0]['text']}\n") print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n") ``` Unload lora0 and replace it with a different adapter: ```python Example theme={null} response = requests.post( url + "/unload_lora_adapter", json={ "lora_name": "lora0", }, ) response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora0", "lora_path": lora0_new, }, ) if response.status_code == 200: print("LoRA adapter loaded successfully.", response.json()) else: print("Failed to load LoRA adapter.", response.json()) ``` Check output again: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0: \n{response.json()[0]['text']}\n") print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` ### OpenAI-compatible API usage You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai\_api\_completions](../basic_usage/openai_api_completions). ### LoRA GPU Pinning Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded. This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported. ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --cuda-graph-max-bs-decode 8 \ --max-loras-per-batch 3 \ --max-lora-rank 256 \ --lora-target-modules all \ --lora-paths \ {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json","pinned":true} \ {"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \ lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora --log-level warning """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter: ```python Example theme={null} response = requests.post( url + "/unload_lora_adapter", json={ "lora_name": "lora1", }, ) response = requests.post( url + "/load_lora_adapter", json={ "lora_name": "lora1", "lora_path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora", "pinned": True, # Pin the adapter to GPU }, ) ``` Verify that the results are expected: ```python Example theme={null} url = f"http://127.0.0.1:{port}" json_data = { "text": [ "List 3 countries and their capitals.", "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": {"max_new_tokens": 32, "temperature": 0}, # The first input uses lora0, and the second input uses lora1 "lora_path": ["lora0", "lora1", "lora2"], } response = requests.post( url + "/generate", json=json_data, ) print(f"Output from lora0 (pinned): \n{response.json()[0]['text']}\n") print(f"Output from lora1 (pinned): \n{response.json()[1]['text']}\n") print(f"Output from lora2 (not pinned): \n{response.json()[2]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` ## Choosing LoRA Backend SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument: * `triton`: Basic Triton-based backend. * `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios. The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend. ```python Example theme={null} server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --lora-backend csgmv \ --max-loras-per-batch 16 \ --lora-paths lora1=path/to/lora1 lora2=path/to/lora2 """ ) ``` ```python Example theme={null} terminate_process(server_process) ``` ## LoRA Overlap Loading By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a \~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks). ```python Example theme={null} lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json" lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora" lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora" server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --enable-lora \ --enable-lora-overlap-loading \ --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \ lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \ lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \ --max-lora-rank 256 \ --max-loras-per-batch 2 \ --max-loaded-loras 4 """ ) url = f"http://127.0.0.1:{port}" wait_for_server(url) ``` ```python Example theme={null} json_data = { "text": [ "Write a very long fairy-tale.", "List 3 countries and their capitals.", "List 3 countries and their capitals.", ], "sampling_params": [ {"max_new_tokens": 1024, "temperature": 0}, {"max_new_tokens": 64, "temperature": 0}, {"max_new_tokens": 64, "temperature": 0}, ], "lora_path": ["lora0", "lora1", "lora2"], } # lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue. # lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will # occur asynchronously and thus decoding for lora0's request won't be blocked. response = requests.post( url + "/generate", json=json_data, ) for i in range(3): print(f"Output from lora{i}: \n{response.json()[i]['text']}\n") ``` ```python Example theme={null} terminate_process(server_process) ``` #### Limitations of LoRA Overlap Loading However, LoRA overlap loading is not free and comes with two important caveats: 1. **Pinned CPU memory requirement**: Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled. 2. **Reduced multi-adapter prefill batching**: With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads). #### Example When Overlap Loading Results in Higher Latency For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms. 1. **Baseline**: The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms` 2. **With LoRA overlap loading enabled**: The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms` In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT. ## Future Works The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development. # Model Loading Source: https://docs.sglang.io/docs/advanced_features/model_loading Control how SGLang loads model weights: load formats, model loader extra config, multithreaded loading, prefetching, and remote/streaming loaders. `--model-path` selects the checkpoint to serve; `--load-format` and the weight-loading flags below control how those weights are read into memory. To stream weights from cloud object storage (S3/GCS/Azure), see [Loading Models from Object Storage](./object_storage). ## How loading works SGLang picks a loader from `--load-format`, falling back to auto-detection from the checkpoint or model path. The default `auto` loader reads `safetensors` and falls back to PyTorch `.bin`. ```bash theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --load-format auto ``` Some formats are auto-detected and override `auto`: * A Mistral native checkpoint is detected and loaded with `mistral`. * A `.gguf` model path is detected and loaded with `gguf`. * An object storage URI (`s3://`, `gs://`, `az://`) is loaded with `runai_streamer`. * A remote URI is loaded with `remote`. ## Load formats Set with `--load-format`:
Format Description
auto Default. Load safetensors if available, otherwise fall back to the PyTorch .bin format.
safetensors Load weights in the safetensors format.
pt Load weights in the PyTorch .bin format.
npcache Load PyTorch-format weights and store a numpy cache to speed up subsequent loads. Only supports .bin checkpoints.
dummy Initialize weights with random values, for profiling.
sharded\_state Each tensor-parallel worker reads only its own pre-sharded shard rather than the full checkpoint, giving a fast load path for large TP models. See examples/runtime/engine/save\_sharded\_state.py for creating a sharded checkpoint.
fastsafetensors Load safetensors using the fastsafetensors iterator.
layered Load weights layer by layer, so a layer can be quantized before the next is loaded, lowering the peak memory envelope.
gguf Load weights in the GGUF format. Auto-detected from a .gguf model path.
bitsandbytes Load weights using bitsandbytes quantization.
mistral Load a Mistral native-format checkpoint. Auto-detected for such checkpoints.
flash\_rl Load a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training. Requires --rl-quant-profile.
runai\_streamer Stream weights from SSDs, shared filesystems, or object storage. See Loading Models from Object Storage.
remote Load tensors from a remote KV/filesystem connector. Auto-detected for remote URIs.
remote\_instance Pull weights over the network from another running SGLang instance (the "seed") rather than from disk. Configured with the --remote-instance-weight-loader-\* flags.
## Model loader extra config `--model-loader-extra-config` takes a JSON string passed to the loader selected by `--load-format`. ```bash theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}' ```
Load format Key Description Default
auto / safetensors / pt / npcache enable\_multithread\_load (bool) Read weight shards with a thread pool instead of sequentially. Disabled by default when --weight-loader-prefetch-checkpoints is set (to avoid I/O oversubscription with the prefetch threads); set this to true to opt back in. true
auto / safetensors / pt / npcache num\_threads (int) Number of worker threads when multithreaded loading is enabled. 8
sharded\_state pattern (str) Filename pattern for per-rank shards. model-rank-\{rank}-part-\{part}.safetensors
fastsafetensors enable\_gds (bool) Use GPU Direct Storage. Set to false when the host does not provide the NVIDIA GPUDirect Storage kernel driver, such as in a gVisor sandbox. true
bitsandbytes qlora\_adapter\_name\_or\_path (str) QLoRA adapter to apply on top of the bitsandbytes-quantized base weights. —
runai\_streamer distributed, concurrency, memory\_limit Streaming controls. See Loading Models from Object Storage. See linked page
## Weight-loading performance flags Top-level arguments that tune how safetensors weights are read, independent of `--load-format`.
Flag Description Default
--download-dir Directory used to download and cache Hugging Face model files. HF default
--weight-loader-disable-mmap Disable mmap while loading safetensors. Can help on filesystems where mmap is slow. off
--weight-loader-prefetch-checkpoints Prefetch checkpoint files into the OS page cache before loading. Each rank prefetches a fraction of the shards, cutting total network I/O on shared filesystems (NFS/Lustre) from N×checkpoint to 1×checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set enable\_multithread\_load=true in --model-loader-extra-config to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op). off
--weight-loader-prefetch-num-threads Threads per rank for checkpoint prefetching. 4
--weight-loader-drop-cache-after-load Call posix\_fadvise(DONTNEED) after successfully loading each shard, freeing page cache. Supported by the standard safetensors and fastsafetensors loaders. off
--custom-weight-loader Import path(s) of a custom weight-loading function, e.g. my\_package.weight\_load\_func. —
## See also * [Loading Models from Object Storage](./object_storage) * [Quantization](./quantization) * [Server Arguments](./server_arguments) # Loading Models from Object Storage Source: https://docs.sglang.io/docs/advanced_features/object_storage SGLang can load models directly from object storage without a full local download. It uses the `runai_streamer` load format to stream model weights from cloud storage, reducing startup time and local storage requirements. ## Overview When loading models from object storage, SGLang uses a two-phase approach: 1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache 2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed ## Supported Storage Backends 1. **Amazon S3**: `s3://bucket-name/path/to/model/` 2. **Google Cloud Storage**: `gs://bucket-name/path/to/model/` 3. **Azure Blob**: `az://some-azure-container/path/` 4. **S3 compatible**: `s3://bucket-name/path/to/model/` ## Quick Start ### Basic Usage Simply provide an object storage URI as the model path: ```bash theme={null} # S3 python -m sglang.launch_server \ --model-path s3://my-bucket/models/llama-3-8b/ \ --load-format runai_streamer # Google Cloud Storage python -m sglang.launch_server \ --model-path gs://my-bucket/models/llama-3-8b/ \ --load-format runai_streamer ``` **Note**: The `--load-format runai_streamer` is automatically detected when using object storage URIs, so you can omit it: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://my-bucket/models/llama-3-8b/ ``` ### With Tensor Parallelism ```bash theme={null} python -m sglang.launch_server \ --model-path gs://my-bucket/models/llama-70b/ \ --tp 4 \ --model-loader-extra-config '{"distributed": true}' ``` ## Configuration ### Load Format The `runai_streamer` load format is designed for object storage, SSDs, and shared filesystems. ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --load-format runai_streamer ``` ### Extended Configuration Parameters Use `--model-loader-extra-config` to pass additional configuration as a JSON string: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --model-loader-extra-config '{ "distributed": true, "concurrency": 8, "memory_limit": 2147483648 }' ``` #### Available Parameters
Parameter Type Description Default
distributed bool Enable distributed streaming for multi-GPU setups. Automatically set to true for object storage paths on CUDA-like devices. Auto-detected
concurrency int Number of concurrent download streams. Higher values can improve throughput for large models. 4
memory\_limit int Memory limit (in bytes) for the streaming buffer. System-dependent
## Performance Considerations ### Distributed Streaming For multi-GPU setups, enable distributed streaming to parallelize weight loading across processes: ```bash theme={null} python -m sglang.launch_server \ --model-path s3://bucket/model/ \ --tp 8 \ --model-loader-extra-config '{"distributed": true}' ``` ## Limitations * **Supported formats**: Only the `.safetensors` weight format is supported. * **Supported devices**: Distributed streaming is supported on CUDA-like devices; otherwise it falls back to non-distributed streaming. ## See Also * [Runai model streamer documentation](https://github.com/run-ai/runai-model-streamer) # Observability Source: https://docs.sglang.io/docs/advanced_features/observability ## Production Metrics SGLang exposes the following metrics via Prometheus. You can enable them by adding `--enable-metrics` when launching the server. You can query them by: ```bash Command theme={null} curl http://localhost:30000/metrics ``` See [Production Metrics](../references/production_metrics) and [Production Request Tracing](../references/production_request_trace) for more details. ## Logging By default, SGLang does not log any request contents. You can log them by using `--log-requests`. You can control the verbosity by using `--log-request-level`. See [Logging](./server_arguments#logging) for more details. You can change verbosity at runtime: ```bash Command theme={null} python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --log-level=debug ``` ## Request Dump and Replay You can dump all requests and replay them later for benchmarking or other purposes. To start dumping, use the following command to send a request to a server: ```bash Command theme={null} python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100 ``` The server will dump the requests into a pickle file for every 100 requests. To replay the request dump, use `scripts/playground/replay_request_dump.py`. ## Crash Dump and Replay Sometimes the server might crash, and you may want to debug the cause of the crash. SGLang can preserve recent request data for replay and collect CUDA device coredumps for low-level debugging. Set the crash diagnostics folder with `--crash-dump-folder /tmp/crash_dump`. When SGLang handles a crash, it writes completed requests retained by the crash-dump buffer plus in-flight requests to `/tmp/crash_dump//crash_dump_.pkl`. The file also contains the server arguments and launch command. Replay it with `scripts/playground/replay_request_dump.py`. On NVIDIA CUDA, the option also sets default environment variables before CUDA initializes. These defaults enable device coredumps on CUDA exceptions and allow SGLang to trigger device coredumps for live scheduler processes when it handles a crash. CUDA device coredumps are written to `/tmp/crash_dump//core.cuda..`. Explicitly configured CUDA coredump environment variables take precedence, including a custom `CUDA_COREDUMP_FILE` path. This option does not configure OS process core dumps. # Advanced Features Source: https://docs.sglang.io/docs/advanced_features/overview Advanced configuration, optimization, and deployment features for SGLang. * [Server Arguments](./server_arguments) * [Session-Aware Radix Cache](./session_radix_cache) * [Hyperparameter Tuning](./hyperparameter_tuning) * [Attention Backend](./attention_backend) * [Speculative Decoding](./speculative_decoding) * [Structured Outputs](./structured_outputs) * [Quantization](./quantization) * [Expert Parallelism](./expert_parallelism) * [Decode Context Parallelism](./dcp) * [LoRA](./lora) * [PD Disaggregation](./pd_disaggregation) * [Pipeline Parallelism](./pipeline_parallelism) * [HiCache](./hicache_best_practices) * [Observability](./observability) * [And more…](./server_arguments) # PD Disaggregation Source: https://docs.sglang.io/docs/advanced_features/pd_disaggregation ## Why and What is PD Disaggregation? Large Language Model (LLM) inference comprises two distinct phases: **Prefill** and **Decode**. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce **Prefill and Decoding (PD) Disaggregation** in SGLang. ### Issues with Unified Scheduling The conventional unified engine, which processes prefill and decode batches together, results in two significant problems: 1. **Prefill Interruption**: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation. 2. **DP Attention Imbalance**: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency. PD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each. For the design details, please refer to [link](https://docs.google.com/document/d/1rQXJwKd5b9b1aOzLh98mnyMhBMhlxXA5ATZTHoQrwvc/edit?tab=t.0). Currently, we support Mooncake and NIXL as the transfer engine. ## Profiling in PD Disaggregation Mode When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options. ## Router Integration For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation). ## Mooncake ### Requirements ```bash theme={null} uv pip install mooncake-transfer-engine ``` ### IB Device Configuration `--disaggregation-ib-device` supports the following formats when using the Mooncake backend: 1. Shared device list for all GPUs: `mlx5_0` or `mlx5_0,mlx5_1` 2. Per-GPU JSON mapping: `{"0": "mlx5_0,mlx5_1", "1": "mlx5_2,mlx5_3"}` 3. Path to a JSON file containing the same per-GPU mapping Each JSON value uses the same comma-separated device list format as the shared configuration. ### Usage ### Llama Single Node ```bash theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --port 30000 \ --disaggregation-ib-device mlx5_roce0 python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --port 30001 \ --base-gpu-id 1 \ --disaggregation-ib-device mlx5_roce0 python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 ``` ### DeepSeek Multi-Node ```bash theme={null} # prefill 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # prefill 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # decode 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 # decode 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-ib-device ${device_name} \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 ``` ### Advanced Configuration PD Disaggregation with Mooncake supports the following environment variables for fine-grained control over system behavior. #### NVLink Transport Configuration To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround. ```bash Command theme={null} export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK export MC_FORCE_MNNVL=True ``` To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP. ```bash theme={null} export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK export MC_INTRANODE_NVLINK=true ``` The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`. #### Prefill Server Configuration
Variable Description Default
**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`** Controls the total number of worker threads for KVCache transfer operations per TP rank A dynamic value calculated by int(0.75 \* os.cpu\_count()) // 8, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions
**`SGLANG_DISAGGREGATION_QUEUE_SIZE`** Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to 1, then we transfer requests one by one according to fcfs strategy `4`
**`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`** Timeout (seconds) for receiving destination KV indices during request initialization `300`
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_ENTRY\_CLEANUP\_INTERVAL Interval (seconds) between cleanups of bootstrap entries 120
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition. Please be aware that this setting will cause prefill instances to take a longer time to clean up the affected memory resources when a running decode node loses connection. #### Decode Server Configuration
Variable Description Default
**`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`** Interval (seconds) between health checks to prefill bootstrap servers `5.0`
**`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`** Consecutive heartbeat failures before marking prefill server offline `2`
**`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`** Timeout (seconds) for receiving KV Cache after request initialization `300`
If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition. ## Heterogeneous TP with GPU Staging Buffer When prefill and decode use different tensor parallelism (TP) sizes (e.g., prefill TP=4, decode DP attention with TP=1), the KV cache memory layout differs between the two sides. The **GPU staging buffer** solves this by gathering KV head slices into a contiguous buffer on the prefill side, performing bulk RDMA transfer, then scattering into the correct KV cache pages on the decode side. This provides **2–5x throughput improvement** over the default per-token slice approach at high concurrency and matches homogeneous TP baselines within \~5%. Enable the staging buffer when prefill and decode use **different TP sizes** with the **Mooncake** transfer backend. When both sides use the same TP size, staging is automatically bypassed even if enabled. > **Note:** The staging buffer is designed for non-MLA models (e.g. GQA, MHA). MLA models (e.g. DeepSeek-V2/V3) should not enable this flag. ### Environment Variables
Variable Description Default
SGLANG\_DISAGG\_STAGING\_BUFFER Enable GPU staging buffer for heterogeneous TP KV transfer False
SGLANG\_DISAGG\_STAGING\_POOL\_SIZE\_MB Decode-side ring buffer pool total size in MB 4096
### Usage Example ```bash Command theme={null} # Set staging buffer environment variables on BOTH prefill and decode export SGLANG_DISAGG_STAGING_BUFFER=1 export SGLANG_DISAGG_STAGING_POOL_SIZE_MB=4096 # Prefill with TP=4 python -m sglang.launch_server \ --model-path $MODEL_PATH \ --disaggregation-mode prefill \ --port 30000 \ --tp 4 \ --trust-remote-code \ --disaggregation-ib-device mlx5_1,mlx5_2 # Decode with TP=1 (or DP attention with effective attention TP=1) python -m sglang.launch_server \ --model-path $MODEL_PATH \ --disaggregation-mode decode \ --port 30001 \ --tp 4 \ --dp 4 \ --enable-dp-attention \ --trust-remote-code \ --disaggregation-ib-device mlx5_3,mlx5_4 # Router python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:30000 \ --decode http://127.0.0.1:30001 \ --host 0.0.0.0 --port 8000 ``` ## NIXL ### Requirements Install via pip. ```bash theme={null} pip install nixl ``` Or build from source - may be required if you already have UCX installed. ```bash theme={null} git clone https://github.com/ai-dynamo/nixl.git cd nixl pip install . --config-settings=setup-args="-Ducx_path=/path/to/ucx" ``` ### Usage ### Llama Single Node ```bash theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --port 30000 \ --disaggregation-transfer-backend nixl python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --port 30001 \ --base-gpu-id 1 \ --disaggregation-transfer-backend nixl python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 ``` ### DeepSeek Multi-Node ```bash theme={null} # prefill 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # prefill 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode prefill \ --host ${local_ip} \ --port 30000 \ --trust-remote-code \ --dist-init-addr ${prefill_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 # decode 0 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 0 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 # decode 1 python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3-0324 \ --disaggregation-transfer-backend nixl \ --disaggregation-mode decode \ --host ${local_ip} \ --port 30001 \ --trust-remote-code \ --dist-init-addr ${decode_master_ip}:5000 \ --nnodes 2 \ --node-rank 1 \ --tp-size 16 \ --dp-size 8 \ --enable-dp-attention \ --moe-a2a-backend deepep \ --mem-fraction-static 0.8 \ --max-running-requests 128 ``` ### Advanced Configuration #### NIXL Backend Selection By default, NIXL uses the **UCX** backend for KV cache transfers. You can select a different NIXL plugin backend depending on your infrastructure using the environment variable `SGLANG_DISAGGREGATION_NIXL_BACKEND`. Example: `export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC` **Available backends:** UCX (default), LIBFABRIC, or any installed NIXL plugin. Example usage: ```bash theme={null} export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend nixl \ --port 30000 ``` ## ASCEND ### Usage Use ascend backend with [memfabric\_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND\_MF\_STORE\_URL being set ```bash Command theme={null} pip install memfabric-hybrid==1.0.0 export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx" ``` Use mooncake backend, more details can be found in mooncake section. ```bash theme={null} export ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true ``` ASCEND\_NPU\_PHY\_ID need to be set in container env ```bash theme={null} export ASCEND_NPU_PHY_ID=xxx ``` ### MIMO Single Node #### prefill ```bash theme={null} # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo export SGLANG_NPU_PROFILING=0 export SGLANG_NPU_PROFILING_STAGE="prefill" export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_BF16_DISPATCH=0 export ASCEND_USE_FIA=1 python3 -m sglang.launch_server \ --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --chunked-prefill-size -1 \ --trust-remote-code --port 10000 \ --host 127.0.0.1 --max-running-requests 16 \ --mem-fraction-static 0.8 \ --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --base-gpu-id 0 \ --disable-radix-cache \ --disable-cuda-graph \ --moe-a2a-backend deepep --deepep-mode normal \ # 2>&1 | tee $SGLANG_LOG_PATH ``` #### decode ```bash theme={null} # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # export ASCEND_LAUNCH_BLOCKING=1 # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256 export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo export SGLANG_NPU_PROFILING=0 export SGLANG_NPU_PROFILING_STAGE="prefill" export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_BF16_DISPATCH=0 export ASCEND_USE_FIA=1 python3 -m sglang.launch_server \ --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ --attention-backend ascend \ --device npu \ --tp-size 8 --nnodes 1 --node-rank 0 \ --trust-remote-code --port 10001 \ --host 127.0.0.1 --max-running-requests 16 \ --mem-fraction-static 0.8 \ --disaggregation-mode decode --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8996 \ --base-gpu-id 8 \ --disable-radix-cache \ --cuda-graph-bs 1 2 4 8 10 12 14 16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --moe-a2a-backend deepep --deepep-mode low_latency \ # 2>&1 | tee $SGLANG_LOG_PATH ``` #### router ```bash theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:10000 \ --decode http://127.0.0.1:10001 \ --host 127.0.0.1 \ --port 9903 \ --health-check-interval-secs 3600 \ --mini-lb \ ``` ### DeepSeek Multi-Node #### Environment ```bash theme={null} echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # IP set to p first node ip export ASCEND_MF_STORE_URL="tcp://XXXXXX:24670" # p node IP P_IP=('XXXXX') # D node IP D_IP=('XXXXX') # enable mlapo export SGLANG_NPU_USE_MLAPO=1 export SGLANG_USE_FIA_NZ=1 export ENABLE_MOE_NZ=1 #export SGLANG_NPU_USE_MULTI_STREAM=1 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" ``` #### prefill ```bash theme={null} MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export HCCL_BUFFSIZE=2600 export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \ --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \ --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ --dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 NODE_RANK=$i break fi done ``` #### decode ```bash theme={null} MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export HCCL_BUFFSIZE=900 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 export TASK_QUEUE_ENABLE=1 export HCCL_SOCKET_IFNAME=data0.3001 export GLOO_SOCKET_IFNAME=data0.3001 python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \ --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \ --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ --load-balance-method round_robin NODE_RANK=$i break fi done ``` #### router ```bash theme={null} python -m sglang_router.launch_router --prefill ${P_IP}:8000 \ --decode ${D_IP}:8001 \ --host ${D_IP} --port 6688 \ --pd-disaggregation \ --health-check-interval-secs 3600 \ ``` # Piecewise CUDA Graph Source: https://docs.sglang.io/docs/advanced_features/piecewise_cuda_graph ## Motivation Standard CUDA graphs capture the entire model forward pass as a single graph. This works well for decode (fixed batch size), but not for extend/prefill where the number of tokens varies across iterations. Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes. Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off. ## Usage PCG is enabled by default for supported configurations. No extra flags needed: ```bash theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct ``` ### Disable PCG ```bash theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disable-piecewise-cuda-graph ``` ### Custom capture sizes ```bash theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --piecewise-cuda-graph-max-tokens 2048 ``` ### Server Args
Argument Default Description
--disable-piecewise-cuda-graph False Disable PCG for extend/prefill.
--enforce-piecewise-cuda-graph False Force-enable PCG, skipping all auto-disable conditions. For testing only.
--piecewise-cuda-graph-max-tokens None (auto) Maximum token count to capture. Defaults to chunked\_prefill\_size (non-MLA) or 2048 (MLA).
--piecewise-cuda-graph-tokens None (auto) Explicit list of token lengths to capture. Auto-generated if not set.
--piecewise-cuda-graph-compiler "eager" Compiler backend for the captured subgraphs. Choices: eager, inductor.
--enable-piecewise-cuda-graph — Deprecated. PCG is now enabled by default. Use --enforce-piecewise-cuda-graph to skip auto-disable conditions.
## Bug Report PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature. ### For Users If you see an error message like the following during server startup, it is a PCG bug: ``` Piecewise CUDA Graph is enabled by default as an experimental feature. To work around this error, add --disable-piecewise-cuda-graph to your launch command. Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose ``` To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include: 1. The full error traceback 2. Model name and quantization method 3. Launch command with all arguments 4. GPU type and driver version ### For Developers Since PCG relies on `torch.compile` to trace the model's forward pass, newly developed CUDA kernels (both JIT kernels and sgl-kernels) are typically not compatible with `torch.compile` out of the box. The tracing will fail on untraceable operations such as JIT compilation, file I/O, or dynamic module loading inside the kernel. To make a kernel compatible with PCG, you need to register it as a custom op using `register_custom_op` from `sglang.srt.utils.custom_op`. This wraps the kernel as an opaque node in the compiled graph so that `torch.compile` will not trace inside it. **Example usage (JIT kernel):** ```python theme={null} from sglang.srt.utils.custom_op import register_custom_op # Inplace operator (no return value) @register_custom_op(mutates_args=["output_q", "output_s"]) def per_token_group_quant_8bit( input: torch.Tensor, output_q: torch.Tensor, output_s: torch.Tensor, ) -> None: # kernel implementation ... ``` **Example usage (operator with output):** ```python theme={null} # out_shape indicates which argument has the same shape as the output @register_custom_op(mutates_args=["x"], out_shape=0) def add(x: torch.Tensor, y: torch.Tensor) -> torch.Tensor: return x.add_(y) ``` For wrapping external library functions (e.g., FlashInfer kernels), use `register_custom_op_from_extern` instead. See `python/sglang/srt/utils/custom_op.py` for full API documentation. ## How it works ### Torch compile backend PCG uses `torch.compile` with a custom backend (`SGLangBackend`) to split and compile the model's forward pass. The flow is: ``` model.forward wrapper → torch.compile(..., backend=SGLangBackend) → FX graph → split_graph() at registered split ops → split_gm (top-level graph that chains the pieces) → replace capturable submodules with CUDAPiecewiseBackend → runtime dispatch: eager split ops + per-piece capture/replay ``` * **Install**: `install_torch_compiled()` replaces `model.forward` with a wrapper function. When `is_in_piecewise_cuda_graph()` returns True, the wrapper dispatches to the compiled callable; otherwise it falls back to the original forward. The first invocation through this path triggers Dynamo tracing and graph compilation — CUDA graph replay only happens after the capture phase completes. * **Split**: When `torch.compile` traces the model, `SGLangBackend` receives the FX graph and calls `split_graph()`. Ops listed in `CompilationConfig.split_ops` are treated as split points, so the graph is cut at each one. These split-op submodules are left to run eagerly at runtime, while the surrounding submodules are compiled and wrapped by `CUDAPiecewiseBackend`. The result is a top-level "stitching graph" (`split_gm`) with children such as `submod_0`, `submod_1`, … interleaving capturable subgraphs and eager split-op submodules. * **Replace**: `PiecewiseCompileInterpreter` iterates over each capturable submodule in `split_gm`, compiles it for general (dynamic) shapes, and replaces it in-place with a `CUDAPiecewiseBackend` instance. Split-op submodules (e.g., attention, all-reduce) are left as-is and run eagerly at runtime. * **Dispatch**: At runtime, calling `split_gm` executes the stitching graph, which calls each submodule in order. Split-op submodules run eagerly. Each `CUDAPiecewiseBackend` submodule goes through three phases: * **Compile warmup** — runs the general-shape compiled path. * **Capture** — for each capture size, runs one warmup pass then records a CUDA graph. * **Steady-state replay** — replays the captured CUDA graph for each forward pass. ### Piecewise cuda graph runner `PiecewiseCudaGraphRunner` orchestrates the full lifecycle through three phases: * **Compile** — Warms up JIT kernels with a dummy forward pass, then wraps the model with `torch.compile`, triggering Dynamo tracing to split the FX graph and create `CUDAPiecewiseBackend` instances for each subgraph piece. * **Capture** — Iterates over capture sizes in reverse order (largest first). For each size, runs the forward pass twice (one warmup, one CUDA graph capture). * **Replay** — At runtime, finds the smallest captured size >= actual token count via binary search, copies inputs into static buffers with zero-padding, replays the captured CUDA graphs, and slices outputs back to the actual token count. ### Memory optimization The memory cost of PCG comes from two parts: **torch memory allocator** and **non-torch memory**. The torch memory allocator overhead is trivial thanks to several optimizations: a global shared memory pool is reused across all CUDA graph runners and capture sizes, capture is done in reverse order (large to small) so smaller graphs reuse memory allocated by larger ones, and output tensors of the last subgraph are stored as weak references to maximize memory reuse. The main memory overhead comes from non-torch memory — the CUDA graph objects themselves require GPU memory to store the recorded kernel launch parameters and internal state. This overhead scales with the number of captured sizes, which is why `piecewise_cuda_graph_max_tokens` is capped conservatively by default. ### Shape configuration Piecewise CUDA graph pre-captures graphs for a set of token counts. At runtime, the actual token count is rounded up to the nearest captured size (via binary search), and the corresponding graph is replayed. If the token count exceeds the largest captured size, the runtime falls back to the normal (non-graph) forward path. The default capture schedule is auto-generated with increasing granularity:
Token range Step size
4 – 32 4
48 – 256 16
288 – 512 32
576 – 1024 64
1280 – 4096 256
4096+ 512
For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround. ## Compatibility PCG is auto-disabled in the following scenarios. We are actively working on expanding compatibility — support for many of these will be coming soon. * Disabled model architectures (e.g., `DeepseekV32ForCausalLM`) * Speculative decoding * DP attention * Pipeline parallelism (`pp_size > 1`) * Non-CUDA hardware (AMD ROCm, Ascend NPU) * MoE A2A backend * LoRA * Multimodal / VLM models * DLLM (diffusion LLM) * Deterministic inference * PD disaggregation * Expert distribution recorder / EPLB Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only). ## Code Reference
File Description
python/sglang/srt/model\_executor/runner\_backend/tc\_piecewise\_cuda\_graph\_backend.py Backend implementation: compile, capture, replay
python/sglang/srt/compilation/compile.py install\_torch\_compiled trampoline
python/sglang/srt/compilation/backend.py SGLangBackend, graph splitting, piecewise compilation
python/sglang/srt/compilation/cuda\_piecewise\_backend.py Per-subgraph CUDA graph capture/replay
python/sglang/srt/compilation/piecewise\_context\_manager.py Global context flags and ForwardContext
python/sglang/srt/compilation/compilation\_config.py Capture sizes, split ops, compiler config
python/sglang/srt/utils/custom\_op.py register\_custom\_op for torch.compile compatibility
python/sglang/srt/server\_args.py Server arguments and auto-disable logic
# Pipeline Parallelism for Long Context Source: https://docs.sglang.io/docs/advanced_features/pipeline_parallelism ## Why Pipeline Parallelism? As Large Language Models (LLMs) scale toward trillion-parameter architectures and "infinite" context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput. Detailed analysis can be found in this [blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/). ## Implementation Refactoring based on Async Communication With Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements. To eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852. The key mechanisms of the implementation include: * **Decoupled Sync/Async Logic in the Event Loop:** The scheduler uses `async_send` in `_pp_send_pyobj_to_next_stage`. Instead of waiting for a transfer to complete, it returns a `P2PWork` handle. The actual synchronization (`P2PWork.work.wait()`) is deferred until `_pp_commit_comm_work` is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight. * **Multi-Stream Execution:** In addition to the main `default_stream`, which serves as the synchronization stream, SGLang utilizes dedicated `forward_stream` and `copy_stream` to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While `_pp_launch_batch` is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch's results using `_pp_process_batch_result`. ## Guidance about Dynamic Chunking ### Why Dynamic Chunking Chunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks. To address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition: Runtime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size) where ***L*** denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length ***L***. Since the computation complexity of the Attention mechanism scales with ***L***, the next chunk size will be progressively reduced as ***L*** grows to maintain an aligned chunk execution time across pipeline stages. Based on this method, the scheduler can predict and dynamically reduce the chunk size during runtime to minimize the bubbles caused by the stage misalignment. To be noticed, the scheduler does not use the raw predicted value. To facilitate efficient KVCache memory management and ensure affinity with hardware execution efficiency, the value is aligned downward to the nearest multiple of max(`--page-size`, 64). ### Chunked Prefill Size and Smoothing Factor When `--enable-dynamic-chunking` is enabled, each chunk size of a sequence is determined dynamically based on the quadratic model that predicts the next chunk size based on the estimated runtime of the initial chunk length. In this case, we use `--chunked-prefill-size` to set up the initial chunk size. When switching to the dynamic chunking mode, the initial chunk size (`--chunked-prefill-size`) should be set to a larger value comparable to the original chunked prefill size, so that there won't be too many chunks. **`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`** is an environmental variable that controls the smoothing factor for the dynamic chunking algorithm, defaulting to 0.75. It determines how much the chunk size can change during the prefill phase. A larger value means a more aggressive chunk size change, which may lead to better performance but also to greater chunk size changes (the chunk size at the end may become very small, which could lead to performance degradation) and more total chunks. When it is set to 1, the chunk size will be adjusted strictly based on the aforementioned quadratic model that predicts the next chunk size. A smaller value means a more conservative chunk size change, which may lead to smaller chunk size changes and fewer total chunks. When it is set to 0, the chunk size will not be adjusted dynamically, so it is identical to the traditional way with a fixed chunked prefill size. Due to the variation in hardware, models, and target workloads, a static configuration is seldom optimal across all scenarios. Consequently, achieving peak performance necessitates a degree of hyperparameter tuning when switching to the dynamic chunking mode. **Tuning Guidance for Dynamic Chunked Prefill** * **Step 1 - Iterate to find the optimal fixed chunked prefill size for the targeted PP size**: Different PP sizes for targeted ITL may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling. * **Step 2 - Initial Chunk Size Selection for Dynamic Chunking**: Set the initial size to 2× or 3× the optimal fixed chunked prefill size. This reduces the total number of chunks and prevents "tail chunks" from underutilizing hardware. To maintain efficiency for extremely large Input Token Lengths (ITL), the dynamic predictor automatically ensures subsequent chunks are at least 1/4 of this initial size. In addition, it is recommended to use a larger initial chunk size (e.g., 4× the optimal fixed chunked prefill size) for such cases as well. * **Step 3 - Smooth Factor Adjustment**: This factor controls how strictly the chunk size adjusts the prediction given by the quadratic performance fitting model. * 1.0: Follows the model strictly. * **0.6 – 0.85 (Recommended)**: Typical range for the best balance between dynamic scaling and hardware stability. Through experiments, we find that a range between 0.6 and 0.85 typically yields the best performance for dynamic chunking. * 0: Disables dynamic adjustment, reverting to traditional fixed-size chunking. * **Another small optimization tip:** Put the larger partition in the higher PP rank when the layers are not evenly divisible across ranks. It can increase the GPU utilization when a larger PP rank is waiting for the previous stage’s result, hence reducing the bubbles on higher PP ranks. If we take DeepSeek-V3.1 as an example, `SGLANG_PP_LAYER_PARTITION=15,15,15,16` usually performs better than `16,15,15,15`. ## Best Practice for Long Context ### Tuning the Chunked Prefill Size Optimizing the chunked prefill size is crucial for balancing pipeline efficiency and resource utilization. The ideal size depends on factors including model architecture, hardware configuration, and typical input lengths. We recommend starting with a small chunk size, such as 4K, and gradually increasing it until you find the optimal size for your specific use case (Different targeted ITL and PP Sizes may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling). Alternatively, you can analyze the hardware capacity and determine the optimal chunk size based on the roofline model. ### Enable Dynamic Chunking and Adjust Smoothing Factor for Ultra-long ITL SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics. ### Case Study on NVIDIA H20 When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-FP8. When enabling dynamic chunking, we first scale the optimal fixed chunked prefill size by a factor of 3 as the initial chunk size. Through experimentation, we found that a multiplier of 2-3 provides an appropriate balance—avoiding excessive initial pipeline bubbles while ensuring that subsequent chunks don't become too small as context length increases. With the default dynamic chunking smoothing factor of 0.75, we performed parameter tuning and determined that a value of 0.65 works optimally with the 12K initial chunk size for the DeepSeek-V3.1, while a value of 0.8 works optimally with the 18K initial chunk size for the Qwen3-235B-A22B-FP8. #### DeepSeek-V3.1 with 128K Input Token Length ```bash Command theme={null} # prefill node 0 (fixed chunked prefill size) python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 4096 ``` ```bash Command theme={null} # prefill node 0 (with dynamic chunking) export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65 python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking ``` #### Qwen3-235B-A22B-FP8 with 128K Input Token Length ```bash Command theme={null} # prefill node 0 (fixed chunked prefill size) python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 6144 ``` ```bash Command theme={null} # prefill node 0 (with dynamic chunking) export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8 python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \ --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \ --port 30000 --dist-init-addr \ --disable-radix-cache --mem-fraction-static 0.8 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \ --max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking ``` Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production. ## Best Practice for Pipeline Parallelism with PD Disaggregation To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation. # Quantization Source: https://docs.sglang.io/docs/advanced_features/quantization SGLang supports various quantization methods, including offline quantization and online dynamic quantization. Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset. Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime. Like NVIDIA FP8 training's [delayed scaling](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#Mixed-precision-training-with-FP8) mechanism, online quantization calculates the appropriate scaling factors on-the-fly to convert high-precision weights into a lower-precision format. **Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.** If you use a pre-quantized model, **do not add `--quantization` to enable online quantization at the same time**. For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer) or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization to guard against abnormal quantization loss regressions. ## Platform Compatibility The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.
Method NVIDIA GPUs AMD GPUs (MI300X/MI325X/MI350X) Ascend NPUs (A2/A3/A5) Notes
fp8 Yes Yes WIP Aiter or Triton backend on AMD
mxfp4 Yes Yes Yes (A5) On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (A5): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline W4A4\_MXFP4 dense and MoE checkpoints (single-level) are auto-detected via modelslim
mxfp8 No No Yes (A5 for Diffusion, LLM Dense Linear and LLM MoE) Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on A5 series; uses CANN npu\_dynamic\_mx\_quant / npu\_quant\_matmul (dense) and npu\_grouped\_matmul\_swiglu\_quant\_v2 / npu\_grouped\_matmul (MoE) kernels
mxfp\_w4a8 No No Yes (A5) Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on A5 series; offline W4A8\_MXFP dense and MoE checkpoints are auto-detected via modelslim
blockwise\_int8 Yes Yes No Triton-based, works on both platforms
w8a8\_int8 Yes Yes No
w8a8\_fp8 Yes Yes No Aiter or Triton FP8 on AMD
awq Yes Yes Yes Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend
gptq Yes Yes Yes Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend
compressed-tensors Yes Yes Partial Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, FP8 not supported yet
quark Yes Yes No AMD Quark quantization; Aiter GEMM paths on AMD
auto-round Yes Yes Partial Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend
quark\_int4fp8\_moe No Yes No AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)
awq\_marlin Yes No No Marlin kernels are CUDA-only
gptq\_marlin Yes No No Marlin kernels are CUDA-only
gguf Yes No Yes CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time
modelopt / modelopt\_fp8 Yes (Hopper/SM90+) No No NVIDIA ModelOpt; requires NVIDIA hardware
modelopt\_fp4 Yes (SM80-SM90 via Marlin; SM100+ native FP4) No No NVIDIA ModelOpt; use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell; supports load-time BF16/FP16/FP8 MoE conversion with per-tensor FP32 activation scales
nvfp4\_online Yes (Blackwell/SM100 or SM103) No No Online MoE-only NVFP4 weight quantization with per-token FP32 activation scales for BF16/FP16/FP8 checkpoints; use modelopt\_fp4 for per-tensor FP32 activation scales; requires flashinfer\_trtllm or flashinfer\_trtllm\_routed
petit\_nvfp4 No Yes (MI250/MI300X/MI325X) No Enables NVFP4 on ROCm via Petit; use modelopt\_fp4 on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See LMSYS blog and AMD ROCm blog.
bitsandbytes Yes Experimental No Depends on bitsandbytes ROCm support
modelslim No No Yes Ascend quantization; Uses CANN kernels
On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details. On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/optimization/quantization) for details. ## GEMM Backends for FP4/FP8 Quantization Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and **NVFP4** GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`. ### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
Backend Hardware Description
auto All Auto-selects based on hardware
deep\_gemm SM90, SM100 JIT-compiled; enabled when DeepGEMM is installed
flashinfer\_trtllm SM100 FlashInfer TensorRT-LLM backend; optimal for low-latency
flashinfer\_cutlass SM100/120 FlashInfer CUTLASS groupwise FP8 GEMM
flashinfer\_deepgemm SM90 Uses swapAB optimization for small M dimensions in decoding
cutlass SM120 sgl-kernel CUTLASS
triton All Fallback; widely compatible
aiter ROCm AMD AITER backend
**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM120); 4) AITER (AMD); 5) Triton (fallback). **MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M. ### `--fp4-gemm-backend` (NVFP4 GEMM)
Backend Hardware Description
auto SM80+ Auto-selects: flashinfer\_cutedsl on SM100; marlin on SM80-SM90; flashinfer\_cutlass otherwise (including SM120)
flashinfer\_cutlass SM100/120 FlashInfer CUTLASS backend
flashinfer\_cudnn SM100/120 (CUDA 13+, cuDNN 9.15+) FlashInfer cuDNN backend
flashinfer\_cutedsl SM100 FlashInfer CuTe DSL backend
flashinfer\_trtllm SM100 FlashInfer TensorRT-LLM backend
marlin SM80-SM90 Weight-only W4A16 fallback for NVFP4 checkpoints
On SM80-SM90, `auto` selects Marlin for NVFP4. NVFP4 GEMM requires FlashInfer to be installed. ## Offline Quantization To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \ --port 30000 --host 0.0.0.0 ``` Take note, if your model is **per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation**, you can opt to include `--quantization w8a8_int8` or `--quantization w8a8_fp8` to invoke the corresponding CUTLASS int8\_kernel or fp8\_kernel in sgl-kernel. This action will ignore the Hugging Face config's quantization settings. For instance, with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`, if you execute with `--quantization w8a8_fp8`, the system will use the `W8A8Fp8Config` from SGLang to invoke the sgl-kernel, rather than the `CompressedTensorsConfig` for vLLM kernels. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \ --quantization w8a8_fp8 \ --port 30000 --host 0.0.0.0 ``` ### Examples of Offline Model Quantization #### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide) We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide). #### Using [auto-round](https://github.com/intel/auto-round) ```bash Command theme={null} # Install pip install auto-round ``` * LLM quantization ```py Example theme={null} # for LLM from auto_round import AutoRound model_id = "meta-llama/Llama-3.2-1B-Instruct" quant_path = "Llama-3.2-1B-Instruct-autoround-4bit" # Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc. scheme = "W4A16" format = "auto_round" autoround = AutoRound(model_id, scheme=scheme) autoround.quantize_and_save(quant_path, format=format) # quantize and save ``` * VLM quantization ```py Example theme={null} # for VLMs from auto_round import AutoRoundMLLM model_name = "Qwen/Qwen2-VL-2B-Instruct" quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit" scheme = "W4A16" format = "auto_round" autoround = AutoRoundMLLM(model_name, scheme) autoround.quantize_and_save(quant_path, format=format) # quantize and save ``` * Command Line Usage (Gaudi/CPU/Intel GPU/CUDA) ```bash Command theme={null} auto-round \ --model meta-llama/Llama-3.2-1B-Instruct \ --bits 4 \ --group_size 128 \ --format "auto_round" \ --output_dir ./tmp_autoround ``` * SGlang API Usage (CPU/CUDA) ```python Example theme={null} from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader from sglang.srt.configs.device_config import DeviceConfig # Configure model with inc quantization and saving model_config = ModelConfig( model_path="meta-llama/Llama-3.2-3B-Instruct", quantization="auto-round-int8", trust_remote_code=True, ) load_config = LoadConfig( inc_save_path="./quantized_model", ) device_config = DeviceConfig(device="cpu") # Load and quantize the model model_loader = get_model_loader(load_config, model_config) quantized_model = model_loader.load_model( model_config=model_config, device_config=device_config, ) ``` * known issues Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative. 1. Mixed-bit Quantization Limitations Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues. 2. Limited Support for Quantized MoE Models Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors. 3. Limited Support for Quantized VLMs Qwen2.5-VL-7B auto\_round:auto\_gptq format: Accuracy is close to zero. GPTQ format: Fails with: ```text Output theme={null} The output size is not aligned with the quantized weight shape ``` auto\_round:auto\_awq and AWQ format: These work as expected. 4. Limited Support for SGlang API Usage SGlang API Usage only supports `auto-round-int8` quantization method now, more quantization methods are on the way. * CPU serving AutoRound INT4 checkpoints (both `auto_round:auto_gptq` and `auto_round:auto_awq` packing formats) can be served on Intel CPUs with AMX support: ```bash theme={null} SGLANG_USE_CPU_ENGINE=1 python3 -m sglang.launch_server \ --model-path OPEA/Qwen2.5-0.5B-Instruct-int4-sym-inc \ --quantization auto-round \ --device cpu --trust-remote-code ``` The current SGLang CPU backend supports only 4-bit AutoRound checkpoints on Intel AMX. Other AutoRound bit-widths and non-AMX CPU backends are not supported by this path. #### Using [GPTQModel](https://github.com/ModelCloud/GPTQModel) ```bash Command theme={null} # install pip install gptqmodel --no-build-isolation -v ``` ```py Example theme={null} from datasets import load_dataset from gptqmodel import GPTQModel, QuantizeConfig model_id = "meta-llama/Llama-3.2-1B-Instruct" quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit" calibration_dataset = load_dataset( "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train" ).select(range(1024))["text"] quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config model = GPTQModel.load(model_id, quant_config) # load model model.quantize(calibration_dataset, batch_size=2) # quantize model.save(quant_path) # save model ``` #### Using [LLM Compressor](https://github.com/vllm-project/llm-compressor/) ```bash Command theme={null} # install pip install llmcompressor ``` Here, we take quantize `meta-llama/Meta-Llama-3-8B-Instruct` to `FP8` as an example to elaborate on how to do offline quantization. ```python Example theme={null} from transformers import AutoTokenizer from llmcompressor.transformers import SparseAutoModelForCausalLM from llmcompressor.transformers import oneshot from llmcompressor.modifiers.quantization import QuantizationModifier # Step 1: Load the original model. MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" model = SparseAutoModelForCausalLM.from_pretrained( MODEL_ID, device_map="auto", torch_dtype="auto") tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) # Step 2: Perform offline quantization. # Step 2.1: Configure the simple PTQ quantization. recipe = QuantizationModifier( targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]) # Step 2.2: Apply the quantization algorithm. oneshot(model=model, recipe=recipe) # Step 3: Save the model. SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic" model.save_pretrained(SAVE_DIR) tokenizer.save_pretrained(SAVE_DIR) ``` Then, you can directly use the quantized model with `SGLang`, by using the following command: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \ --port 30000 --host 0.0.0.0 ``` #### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. **Offline vs. Online Quantization:** SGLang supports two modes for ModelOpt. * **Offline Quantization (pre-quantized):** * **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint. * **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage. * **Cons:** Requires an extra preparation step. * **Online Quantization (quant and serve):** * **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*. * **Pros:** Convenient (no new checkpoint needed). * **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM). The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints. ##### Using Pre-Quantized Checkpoints If a model is already quantized (e.g., from Hugging Face), you can load it directly. * **FP8 Models:** Use `--quantization modelopt_fp8`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \ --quantization modelopt_fp8 \ --port 30000 ``` * **FP4 Models:** Use `--quantization modelopt_fp4`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \ --quantization modelopt_fp4 \ --port 30000 ``` ##### Creating Your Own Quantized Checkpoints If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script. **Why quantize?** * Reduce VRAM usage * Higher throughput and lower latency * More flexible deployment (on smaller GPUs) **What can be quantized?** * The entire model * MLP layers only * KV cache **Key options in `hf_ptq.py`:** `--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only` `--kv_cache_qformat`: KV cache quantization format (default: `fp8`) **Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly. **Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time. For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq). SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment. ##### Installation First, install ModelOpt: ```bash Command theme={null} pip install nvidia-modelopt ``` ##### Quantization and Export Workflow SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt\_quantize\_and\_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)): ```bash Command theme={null} # Quantize and export a model using ModelOpt FP8 quantization python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ --export-dir ./quantized_tinyllama_fp8 \ --quantization-method modelopt_fp8 # For FP4 quantization (requires Blackwell GPU) python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ --export-dir ./quantized_tinyllama_fp4 \ --quantization-method modelopt_fp4 ``` ##### Available Quantization Methods * `modelopt_fp8`: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs * `modelopt_fp4`: FP4 quantization with optimal performance on Nvidia Blackwell GPUs ##### Python API Usage You can also use ModelOpt quantization programmatically: ```python Example theme={null} import sglang as sgl from sglang.srt.configs.device_config import DeviceConfig from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader # Configure model with ModelOpt quantization and export model_config = ModelConfig( model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0", quantization="modelopt_fp8", # or "modelopt_fp4" trust_remote_code=True, ) load_config = LoadConfig( modelopt_export_path="./exported_model", modelopt_checkpoint_save_path="./checkpoint.pth", # optional, fake quantized checkpoint ) device_config = DeviceConfig(device="cuda") # Load and quantize the model (export happens automatically) model_loader = get_model_loader(load_config, model_config) quantized_model = model_loader.load_model( model_config=model_config, device_config=device_config, ) ``` ##### Deploying Quantized Models After quantization and export, you can deploy the model with SGLang: ```bash Command theme={null} # Deploy the exported quantized model python -m sglang.launch_server \ --model-path ./quantized_tinyllama_fp8 \ --quantization modelopt \ --port 30000 --host 0.0.0.0 ``` Or using the Python API (use the same path as `modelopt_export_path` from the quantize step): ```python Example theme={null} import sglang as sgl def main(): # Deploy exported ModelOpt quantized model # Path must match modelopt_export_path from quantize step (e.g., ./exported_model) llm = sgl.Engine( model_path="./exported_model", quantization="modelopt", ) # Run inference prompts = [ "Hello, how are you?", "What is the capital of France?", ] sampling_params = { "temperature": 0.8, "top_p": 0.95, "max_new_tokens": 100, } outputs = llm.generate(prompts, sampling_params) for i, output in enumerate(outputs): print(f"Prompt: {prompts[i]}") print(f"Output: {output['text']}") if __name__ == "__main__": main() ``` ##### Advanced Features **Checkpoint Management**: Save and restore fake quantized checkpoints for reuse: ```bash Command theme={null} # Save the fake quantized checkpoint during quantization python examples/usage/modelopt_quantize_and_export.py quantize \ --model-path meta-llama/Llama-3.2-1B-Instruct \ --export-dir ./quantized_model \ --quantization-method modelopt_fp8 \ --checkpoint-save-path ./my_checkpoint.pth # The checkpoint can be reused for future quantization runs and skip calibration ``` **Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API: ```python Example theme={null} from sglang.srt.configs.device_config import DeviceConfig from sglang.srt.configs.load_config import LoadConfig from sglang.srt.configs.model_config import ModelConfig from sglang.srt.model_loader.loader import get_model_loader model_config = ModelConfig( model_path="meta-llama/Llama-3.2-1B-Instruct", quantization="modelopt_fp8", trust_remote_code=True, ) load_config = LoadConfig( modelopt_checkpoint_restore_path="./my_checkpoint.pth", modelopt_export_path="./exported_model", ) # Load and export the model (DeviceConfig defaults to device="cuda") model_loader = get_model_loader(load_config, model_config) model_loader.load_model(model_config=model_config, device_config=DeviceConfig()) ``` ##### Benefits of ModelOpt * **Hardware Optimization**: Specifically optimized for NVIDIA GPU architectures * **Advanced Quantization**: Supports cutting-edge FP8 and FP4 quantization techniques * **Seamless Integration**: Automatic export to HuggingFace format for easy deployment * **Calibration-based**: Uses calibration datasets for optimal quantization quality * **Production Ready**: Enterprise-grade quantization with NVIDIA support #### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim) MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. * **Installation** ```bash Command theme={null} # Clone repo and install msmodelslim: git clone https://gitcode.com/Ascend/msmodelslim.git cd msmodelslim bash install.sh ``` * **LLM quantization** Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card). > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). *Traditional quantification methods require the preparation of calibration data files (`.jsonl` formats) for calibration in the quantification process.* ````bash Command theme={null} Qwen3-32B/ # floating-point model downloaded from official HF (or modelscope) repo msmodelslim/ # msmodelslim repo |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones) |----- some file (such as laos_calib.jsonl) |----- lab_practice # best practice folder with configs for quantization |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config |----- another folders output_folder/ # generated by below command |----- quant_model_weights-00001-of-0001.safetensors # quantized weights |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.) |----- another files (such as config.json, tokenizer.json, etc.) ```` Run quantization using one-click quantization (recommended): ```bash Command theme={null} msmodelslim quant \ --model_path ${MODEL_PATH} \ --save_path ${SAVE_PATH} \ --device npu:0,1 \ --model_type Qwen3-32B \ --quant_type w8a8 \ --trust_remote_code True ``` * **Usage Example** ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path $PWD/Qwen3-32B-w8a8 \ --port 30000 --host 0.0.0.0 ``` * **Available Quantization Methods**: * [x] `W4A4_DYNAMIC` linear with online quantization of activations * [x] `W8A8` linear with offline quantization of activations * [x] `W8A8_DYNAMIC` linear with online quantization of activations * [x] `W4A4_DYNAMIC` MOE with online quantization of activations * [x] `W4A4_MXFP4` MOE with dynamic MXFP4 activation quantization * [x] `W4A8_DYNAMIC` MOE with online quantization of activations * [x] `W4A8_MXFP` MOE with dynamic MXFP8 activation quantization * [x] `W8A8_DYNAMIC` MOE with online quantization of activations * [ ] `W4A8` linear TBD * [ ] `W4A16` linear TBD * [ ] `W48A16` linear TBD * [ ] `W4A16` MoE in progress * [ ] `W8A16` MoE in progress * [ ] `KV Cache` in progress * [ ] `Attention` in progress For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo. ## Online Quantization To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --quantization fp8 \ --port 30000 --host 0.0.0.0 ``` Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to `["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"]`. ### `nvfp4_online` online quantization method Use `--quantization nvfp4_online` to convert eligible BF16, FP16, or FP8 MoE expert weights to NVFP4 at load time with per-token FP32 activation scales. Use `modelopt_fp4` for serialized NVFP4 checkpoints or the same load-time conversion with per-tensor FP32 activation scales. The design separates weight quantization from activation scaling: * **Weights:** SGLang quantizes each eligible MoE expert weight tensor as it is loaded, using standard 2D NVFP4 weight quantization. The generated NVFP4 weights use static E4M3 block scales plus static per-tensor FP32 scales derived from the weight amax. For gated MoE experts, the w1/w3 pair shares one per-tensor FP32 scale. * **Activations:** FlashInfer computes and propagates one FP32 scale per token at runtime. Backends that accept one per-tensor FP32 activation scale must use `modelopt_fp4`. * **FP8 checkpoints:** If an eligible expert weight is stored as FP8, SGLang first dequantizes that tensor with the checkpoint scale and then requantizes it to NVFP4 during loading. * **Other layers:** Dense linear layers stay in their source checkpoint precision or checkpoint quantization path. Only `--moe-runner-backend flashinfer_trtllm` and `--moe-runner-backend flashinfer_trtllm_routed` are supported. If `--moe-runner-backend` is omitted, SGLang selects `flashinfer_trtllm`. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor's per-tensor amax-derived FP32 scale. FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. Both online modes honor `SGLANG_FP4_IGNORED_LAYERS`; for FP8 source checkpoints, listed experts remain FP8 instead of being converted to NVFP4. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \ --tp-size 2 \ --ep-size 2 \ --quantization nvfp4_online \ --port 30000 --host 0.0.0.0 ``` ### `quark_int4fp8_moe` online quantization method SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8. Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly. ### `quark_mxfp4` online quantization method SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 or NVFP4 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs. Example (BF16 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path Qwen/Qwen3-30B-A3B \ --tensor-parallel-size 1 \ --quantization quark_mxfp4 ``` #### Online NVFP4 to MXFP4 Requantization The option `--quantization quark_mxfp4` supports converting NVFP4 checkpoints (e.g. `nvidia/Kimi-K2.6-NVFP4`) to MXFP4 at load time to allow efficient inference using supported AMD hardware (gfx95x+): * The quantization metadata of the source NVFP4 checkpoint is read from either `config.json` (`quantization_config`) or a standalone `hf_quant_config.json`; * Producer-declared excluded modules will remain in higher precision; * NVFP4 checkpoints with mixed precision (`"quant_algo": "MIXED_PRECISION"`, e.g. `nvidia/Qwen3.5-397B-A17B-NVFP4-V2`) are also supported. Example (NVFP4 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path nvidia/Kimi-K2.6-NVFP4 \ --tensor-parallel-size 4 \ --quantization quark_mxfp4 \ ``` #### Online FP8 to MXFP4 Requantization The option `--quantization quark_mxfp4` supports converting FP8 dense and MOE models to MXFP4, following this logic: 1. Load an FP8 weight tensor, 2. Dequantize it to BF16, 3. Requantize it to MXFP4 progressively during weight loading. Example (FP8 to MXFP4 requantization): ```bash theme={null} sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \ --tensor-parallel-size 1 \ --quantization quark_mxfp4 ``` ### Intel® Neural Compressor online quantization method SGLang supports quantization methods based on the advanced algorithm [auto-round](https://github.com/intel/auto-round) in [Intel® Neural Compressor](https://github.com/intel/neural-compressor). You can simply specify `--quantization auto-round-int8` to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way. ##### Available Quantization Methods | Quantization Method | Schemes | Validated Hardware Environment | | :------------------ | :----------------------------------------------------------------------------------- | :--------------------------------------------------- | | auto-round-int8 | INT8 per-channel quantized weight
INT8 per-token dynamic quantized activation | Intel Xeon Scalable processor
Nvidia A100 GPU | ## Diffusion Model Quantization on Ascend NPU SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend A5 NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI. **Requirements:** Ascend A5, CANN ≥ 8.0.RC3 ### Online MXFP8 Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --num-gpus 4 ``` ### Offline MXFP8 (ModelSlim) Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`: ```bash theme={null} sglang generate \ --model-path /path/to/wan2_2_mxfp8_diffusers \ --prompt "a beautiful sunset" \ --save-output ``` For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/optimization/quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim). ## Reference * [GPTQModel](https://github.com/ModelCloud/GPTQModel) * [LLM Compressor](https://github.com/vllm-project/llm-compressor/) * [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer) * [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq) * [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html) * [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/) * [auto-round](https://github.com/intel/auto-round) * [ModelSlim](https://gitcode.com/Ascend/msmodelslim) # Quantized KV Cache Source: https://docs.sglang.io/docs/advanced_features/quantized_kv_cache Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences. Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used. **Performance Warning**: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits. **Backend Support**: Not all attention backends support quantized KV cache. Refer to [Attention Backend](./attention_backend) for which backends support it. ## Supported Formats SGLang supports the following quantized KV cache formats: ### FP8 Format [OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point formats: * **E5M2** (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision * **E4M3** (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0) ### FP4 Format FP4 quantization is currently experimental. [OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format: * **E2M1** (1 sign bit, 2 exponent bits, 1 mantissa bit): Uses block-based microscaling where tensors are divided into blocks of consecutive elements, with each block sharing a single 8-bit exponential scaling factor. While OCP specifies blocks of 32 elements, SGLang's current implementation uses blocks of 16 elements for KV cache quantization. ## Usage ### Enabling Quantized KV Cache To enable quantized KV cache, use the `--kv-cache-dtype` argument when launching the server: ```bash Command theme={null} # Enable FP8 E5M2 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp8_e5m2 \ # Enable FP8 E4M3 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp8_e4m3 \ # Enable NVFP4 FP4 E2M1 KV cache python3 -m sglang.launch_server \ --model-path nvidia/DeepSeek-R1-0528-NVFP4 \ --kv-cache-dtype nvfp4 \ # Enable block-size-16 FP4 E2M1 KV cache python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-0528 \ --kv-cache-dtype fp4_mx_block16 \ ``` ### Scaling Factors FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache. Currently, only per-tensor (scalar) scaling factors are supported. Scaling factors can be: * **Loaded from checkpoints**: Pre-quantized models (e.g., ModelOpt) may include `k_scale` and `v_scale` parameters that are automatically loaded * **Provided via JSON**: Supply scaling factors via `--quantization-param-path`. The JSON file should follow this format: ```json Config theme={null} { "kv_cache": { "dtype": "float8_e4m3fn", "scaling_factor": { "0": { "0": 1.0, "1": 1.0 } } } } ``` Where the outer keys in `scaling_factor` are tensor parallel ranks and inner keys are layer indices. If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues. **FP4 (MXFP4)**: Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed. ## Performance Considerations ### Memory Savings Quantized KV cache provides significant memory savings: * **BF16 → FP4**: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead) FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios. This enables longer context lengths or more concurrent requests within the same memory budget. ### Accuracy Impact #### FP8 Accuracy FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2). #### FP4 Accuracy FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from [PR #10078](https://github.com/sgl-project/sglang/pull/10078) (MLA) and [PR #12612](https://github.com/sgl-project/sglang/pull/12612) (MHA) show: **Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528)** On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:
Model Dataset KV16 KV8 (FP8 E4M3) KV4 (FP4 E2M1)
Qwen3-235B-A22B gsm8k 0.9168 0.9181 0.9186
Qwen3-235B-A22B aime25 0.7733 0.7333 0.6000
Qwen3-235B-A22B gpqa\_diamond 0.7010 0.6899 0.6778
DeepSeek-R1-0528 gsm8k 0.9157 0.9154 0.9124
DeepSeek-R1-0528 aime25 0.5067 0.4934 0.4000
DeepSeek-R1-0528 gpqa\_diamond 0.7707 0.7697 0.7273
**Smaller Models (e.g., GPT-OSS-120B)** On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:
Model Dataset KV16 KV8 (FP8 E4M3) KV4 (FP4 E2M1)
GPT-OSS-120B gsm8k 0.9161 0.9163 0.9152
GPT-OSS-120B aime25 0.7533 0.7667 0.3533
GPT-OSS-120B gpqa\_diamond 0.5081 0.5434 0.3202
**Key Observations:** * **Simple datasets (e.g., gsm8k)**: FP4 maintains accuracy close to FP8/BF16 across model sizes * **Model size matters**: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models * **Context length**: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant. Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy. ## Best Practices * **Use pre-quantized models**: Prefer models quantized offline with scaling factors included in the checkpoint. * **Choose the right format**: Use `fp8_e4m3` for better accuracy (recommended), `fp8_e5m2` for larger dynamic range, or `nvfp4` / `fp4_mx_block16` for maximum memory savings (experimental) * **Check backend compatibility**: Verify that your chosen attention backend supports quantized KV cache See also: * [Quantization](./quantization) * [Attention Backend](./attention_backend) * [Server Arguments](./server_arguments) # Reasoning-Aware Compression Source: https://docs.sglang.io/docs/advanced_features/reasoning_aware_compression Pruning a reasoning model with a standard calibration set does more damage than pruning a conventional LLM — and it can make the model *slower*. Reasoning-Aware Compression (RAC) fixes this by changing what the pruning solver calibrates on: the model's own chain of thought, generated with SGLang. From [*Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction*](https://arxiv.org/abs/2509.12464) (ICLR 2026). ## The problem One-shot pruning methods such as SparseGPT and Wanda choose which weights to remove by minimizing a layer-wise reconstruction error against a calibration activation matrix `X`: ``` min_{W'} || W X - W' X ||_F^2 s.t. ||W'||_0 <= S ``` `X` is conventionally built from **prompt** tokens — a slice of C4, or a set of task prompts. That is a fair proxy for a typical serving workload, where the prompt dominates the token count. Reasoning models invert that ratio. They emit thousands of chain-of-thought tokens per query, so almost every forward pass the pruned model will ever run is over a token it generated itself. Calibrating only on prompts optimizes the pruned weights for a distribution the model barely visits. The result is not a graceful accuracy decay. The pruned model starts to ramble: it produces longer chains of thought *and* answers less accurately, so pruning increases end-to-end latency instead of reducing it. At 50% sparsity on MATH-500, C4-calibrated DeepSeek-R1-Distill-Qwen-7B takes almost six times as long to evaluate as the dense model it was meant to accelerate. ## The fix RAC samples the dense model's own on-policy rollout during calibration and reconstructs the prompt and decode activations jointly: ``` X_RAC = [ X_prompt , X_decode ] ``` The solver is untouched, so this is a drop-in change to any existing SparseGPT or Wanda workflow. DeepSeek-R1-Distill-Qwen-7B, MATH-500, SparseGPT at 50% sparsity, 1M calibration tokens: | Calibration set | acc\@1 | Eval wall clock | | --------------------------------- | --------- | --------------- | | Dense (no pruning) | 0.936 | 23.3 min | | C4 | 0.744 | 135.0 min | | Task prompts only | 0.812 | 115.6 min | | **RAC (prompts + on-policy CoT)** | **0.900** | **35.3 min** | Across DeepSeek-R1-Distill-Qwen (1.5B–32B) and Qwen3 (1.7B–14B), the paper reports that RAC keeps up to 95% of dense accuracy at 50% sparsity, improving on prompt-only calibration by up to 17 points. ## Using it SGLang ships the recipe as a runnable example at [`examples/usage/reasoning_aware_compression`](https://github.com/sgl-project/sglang/tree/main/examples/usage/reasoning_aware_compression), in three phases: | Phase | Script | What it does | | ----- | ----------------------- | ---------------------------------------------------------------- | | I | `rac_collect_traces.py` | `sgl.Engine` samples on-policy CoT traces into a calibration set | | II | `rac_prune.py` | `llm-compressor` runs SparseGPT/Wanda against those activations | | III | `rac_serve_and_eval.py` | SGLang serves the sparse checkpoint and scores MATH-500 | Phase I is the expensive step — the paper's budget is 1M on-policy CoT tokens — and is where SGLang's batched generation does the work. Phase II delegates the pruning solver to [`llm-compressor`](https://github.com/vllm-project/llm-compressor), which is **not** an SGLang dependency; install it separately with `pip install "llmcompressor>=0.12.0"`. ```bash theme={null} cd examples/usage/reasoning_aware_compression python rac_collect_traces.py \ --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --dataset open-r1/OpenR1-Math-220k --prompt-column problem \ --target-tokens 1000000 --output-dir ./rac_traces_math python rac_prune.py \ --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ --calibration ./rac_traces_math/traces.jsonl \ --sparsity 0.5 --output-dir ./rac_pruned_50 python -m sglang.launch_server --model-path ./rac_pruned_50 ``` The example README walks through building the paper's prompt-only baseline from the same prompts so you can compare calibration strategies head to head. ## Evaluating a pruned reasoning model Accuracy alone will hide the failure mode described above. Always report **mean completion length** and **wall clock** alongside accuracy when comparing pruned reasoning checkpoints — a model that scores two points lower while emitting three times the chain of thought is not a good trade. `rac_serve_and_eval.py` reports all three. ## Related * [Quantization](/docs/advanced_features/quantization) — the other axis of model compression, applied at serving time. # Reasoning Parser Source: https://docs.sglang.io/docs/advanced_features/separate_reasoning SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). ## Supported Models & Parsers
Model Reasoning tags Parser Notes
[Apertus 2509 models](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509) `<|inner_prefix|>` … `<|inner_suffix|>` `apertus2509` Supports `enable_thinking` parameter
[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d) `` … `` `deepseek-r1` Supports all variants (R1, R1-0528, R1-Distill)
[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) `` … `` `deepseek-v3` Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter
[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) `` … `` `qwen3` Supports `enable_thinking` parameter
[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507) `` … `` `qwen3` or `qwen3-thinking` Always generates thinking content
[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) `◁think▷` … `◁/think▷` `kimi_k2` Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.
[GPT OSS](https://huggingface.co/openai/gpt-oss-120b) `<|channel|>analysis<|message|>` … `<|end|>` `gpt-oss` N/A
### Model-Specific Behaviors **Apertus 2509:** * Uses `<|inner_prefix|>` and `<|inner_suffix|>` to delimit reasoning content. For agentic tool use, also specify `--tool-call-parser apertus2509`. **DeepSeek-R1 Family:** * DeepSeek-R1: No `` start tag, jumps directly to thinking content * DeepSeek-R1-0528: Generates both `` start and `` end tags * Both are handled by the same `deepseek-r1` parser **DeepSeek-V3 Family:** * DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`) **Qwen3 Family:** * Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates * Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks **Kimi K2:** * Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`. **GPT OSS:** * GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags ## Usage ### Launching the Server Specify the `--reasoning-parser` option. ```python Example theme={null} import requests from openai import OpenAI from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning" ) wait_for_server(f"http://localhost:{port}") ``` Note that `--reasoning-parser` defines the parser used to interpret responses. ### OpenAI Compatible API Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1: * `reasoning_content`: The content of the CoT. * `content`: The content of the final answer. ```python Example theme={null} # Initialize OpenAI-like client client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1") model_name = client.models.list().data[0].id messages = [ { "role": "user", "content": "What is 1+3?", } ] ``` #### Non-Streaming Request ```python Example theme={null} response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=False, # Non-streaming extra_body={"separate_reasoning": True}, ) print_highlight("==== Reasoning ====") print_highlight(response_non_stream.choices[0].message.reasoning_content) print_highlight("==== Text ====") print_highlight(response_non_stream.choices[0].message.content) ``` #### Streaming Request ```python Example theme={null} response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=True, # Non-streaming extra_body={"separate_reasoning": True}, ) reasoning_content = "" content = "" for chunk in response_stream: if chunk.choices[0].delta.content: content += chunk.choices[0].delta.content if chunk.choices[0].delta.reasoning_content: reasoning_content += chunk.choices[0].delta.reasoning_content print_highlight("==== Reasoning ====") print_highlight(reasoning_content) print_highlight("==== Text ====") print_highlight(content) ``` Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content). ```python Example theme={null} response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=True, # Non-streaming extra_body={"separate_reasoning": True, "stream_reasoning": False}, ) reasoning_content = "" content = "" for chunk in response_stream: if chunk.choices[0].delta.content: content += chunk.choices[0].delta.content if chunk.choices[0].delta.reasoning_content: reasoning_content += chunk.choices[0].delta.reasoning_content print_highlight("==== Reasoning ====") print_highlight(reasoning_content) print_highlight("==== Text ====") print_highlight(content) ``` The reasoning separation is enable by default when specify . **To disable it, set the `separate_reasoning` option to `False` in request.** ```python Example theme={null} response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0.6, top_p=0.95, stream=False, # Non-streaming extra_body={"separate_reasoning": False}, ) print_highlight("==== Original Output ====") print_highlight(response_non_stream.choices[0].message.content) ``` ### SGLang Native API ```python Example theme={null} from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) gen_url = f"http://localhost:{port}/generate" gen_data = { "text": input, "sampling_params": { "skip_special_tokens": False, "max_new_tokens": 1024, "temperature": 0.6, "top_p": 0.95, }, } gen_response = requests.post(gen_url, json=gen_data).json()["text"] print_highlight("==== Original Output ====") print_highlight(gen_response) parse_url = f"http://localhost:{port}/separate_reasoning" separate_reasoning_data = { "text": gen_response, "reasoning_parser": "deepseek-r1", } separate_reasoning_response_json = requests.post( parse_url, json=separate_reasoning_data ).json() print_highlight("==== Reasoning ====") print_highlight(separate_reasoning_response_json["reasoning_text"]) print_highlight("==== Text ====") print_highlight(separate_reasoning_response_json["text"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ### Offline Engine API ```python Example theme={null} import sglang as sgl from sglang.srt.parser.reasoning_parser import ReasoningParser from sglang.utils import print_highlight llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) sampling_params = { "max_new_tokens": 1024, "skip_special_tokens": False, "temperature": 0.6, "top_p": 0.95, } result = llm.generate(prompt=input, sampling_params=sampling_params) generated_text = result["text"] # Assume there is only one prompt print_highlight("==== Original Output ====") print_highlight(generated_text) parser = ReasoningParser("deepseek-r1") reasoning_text, text = parser.parse_non_stream(generated_text) print_highlight("==== Reasoning ====") print_highlight(reasoning_text) print_highlight("==== Text ====") print_highlight(text) ``` ```python Example theme={null} llm.shutdown() ``` ## Supporting New Reasoning Model Schemas For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly. # Server Arguments Source: https://docs.sglang.io/docs/advanced_features/server_arguments This page provides a list of server arguments used in the command line to configure the behavior and performance of the language model server during deployment. These arguments enable users to customize key aspects of the server, including model selection, parallelism policies, memory management, and optimization techniques. You can find all arguments by `python3 -m sglang.launch_server --help` ## Common launch commands * To use a configuration file, create a YAML file with your server arguments and specify it with `--config`. CLI arguments will override config file values. ```bash Command theme={null} # Create config.yaml cat > config.yaml << EOF model-path: meta-llama/Meta-Llama-3-8B-Instruct host: 0.0.0.0 port: 30000 tensor-parallel-size: 2 enable-metrics: true log-requests: true EOF # Launch server with config file python -m sglang.launch_server --config config.yaml ``` * To enable multi-GPU tensor parallelism, add `--tp 2`. If it reports the error "peer access is not supported between these two devices", add `--enable-p2p-check` to the server launch command. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 2 ``` * To enable multi-GPU data parallelism, add `--dp 2`. Data parallelism is better for throughput if there is enough memory. It can also be used together with tensor parallelism. The following command uses 4 GPUs in total. We recommend [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for data parallelism. ```bash Command theme={null} python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --dp 2 --tp 2 ``` * If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --mem-fraction-static 0.7 ``` * See [hyperparameter tuning](./hyperparameter_tuning) on tuning hyperparameters for better performance. * For docker and Kubernetes runs, you need to set up shared memory which is used for communication between processes. See `--shm-size` for docker and `/dev/shm` size update for Kubernetes manifests. * If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096 ``` * To enable fp8 weight quantization, add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments. * To enable fp8 kv cache quantization, add `--kv-cache-dtype fp8_e4m3` or `--kv-cache-dtype fp8_e5m2`. * To enable deterministic inference and batch invariant operations, add `--enable-deterministic-inference`. More details can be found in [deterministic inference document](./deterministic_inference). * If a multimodal server accepts requests from untrusted clients, restrict remote image, video, and audio URLs with `--allowed-media-domains`. SGLang checks the initial URL and every redirect destination against the exact-hostname allowlist. Remote media downloads are limited to 64 MiB by default; adjust `--media-url-max-file-size-mb` when larger trusted media is required. ```bash Command theme={null} python -m sglang.launch_server \ --model-path Qwen/Qwen2.5-VL-7B-Instruct \ --allowed-media-domains upload.wikimedia.org raw.githubusercontent.com ``` Without `--allowed-media-domains`, HTTP(S) media from any domain remains allowed for backward compatibility. Do not expose that configuration to untrusted users. Local paths and `data:` URLs are not governed by the domain allowlist. * To enable decode context parallelism for MLA models, add `--dcp-size N`. See [Decode Context Parallelism](./dcp). * If the model does not have a chat template in the Hugging Face tokenizer, you can specify a [custom chat template](../references/custom_chat_template). If the tokenizer has multiple named templates (e.g., 'default', 'tool\_use'), you can select one using `--hf-chat-template-name tool_use`. * To run tensor parallelism on multiple nodes, add `--nnodes 2`. If you have two nodes with two GPUs on each node and want to run TP=4, let `sgl-dev-0` be the hostname of the first node and `50000` be an available port, you can use the following commands. If you meet deadlock, please try to add `--disable-cuda-graph` * (Note: This feature is out of maintenance and might cause error) To enable `torch.compile` acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. By default, the cache path is located at `/tmp/torchinductor_root`, you can customize it using environment variable `TORCHINDUCTOR_CACHE_DIR`. For more details, please refer to [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [Enabling cache for torch.compile](../references/torch_compile_cache). ```bash Command theme={null} # Node 0 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --tp 4 \ --dist-init-addr sgl-dev-0:50000 \ --nnodes 2 \ --node-rank 0 # Node 1 python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --tp 4 \ --dist-init-addr sgl-dev-0:50000 \ --nnodes 2 \ --node-rank 1 ``` Please consult the documentation below and [server\_args.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py) to learn more about the arguments you may provide when launching a server. ## Model and tokenizer
Argument Description Defaults Options
`--model-path`
`--model`		 The path of the model weights. This can be a local folder or a Hugging Face repo ID. ` None` Type: str
`--tokenizer-path` The path of the tokenizer. ` None` Type: str
`--tokenizer-mode` Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. `auto` auto, slow
`--tokenizer-backend` Tokenizer backend. 'huggingface' uses the default HuggingFace tokenizers library; 'fastokens' uses the fastokens library for faster tokenization. Requires the fastokens package to be installed. `huggingface` huggingface, fastokens
`--tokenizer-worker-num` The worker num of the tokenizer manager. `1` Type: int
`--skip-tokenizer-init` If set, skip init tokenizer and pass input\_ids in generate request. `False` bool flag (set to enable)
`--load-format` The format of the model weights to load. "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. "pt" will load the weights in the pytorch bin format. "safetensors" will load the weights in the safetensors format. "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. "dummy" will initialize the weights with random values, which is mainly for profiling."gguf" will load the weights in the gguf format. "bitsandbytes" will load the weights using bitsandbytes quantization."layered" loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. "flash\_rl" will load the weights in flash\_rl format. "fastsafetensors" and "private" are also supported. "runai\_streamer" enables direct model loading from object storage and shared file systems. `auto` auto, pt, safetensors, npcache, dummy, sharded\_state, gguf, bitsandbytes, mistral, layered, flash\_rl, remote, remote\_instance, fastsafetensors, private, runai\_streamer
`--model-loader-extra-config` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. `{}` Type: str
`--trust-remote-code` Whether or not to allow for custom models defined on the Hub in their own modeling files. `False` bool flag (set to enable)
`--context-length` The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). ` None` Type: int
`--is-embedding` Whether to use a CausalLM as an embedding model. `False` bool flag (set to enable)
`--enable-multimodal` Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen. ` None` bool flag (set to enable)
`--revision` The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. ` None` Type: str
`--model-impl` Which implementation of the model to use. "auto" will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. "sglang" will use the SGLang model implementation. "transformers" will use the Transformers model implementation. `auto` Type: str
`--detokenizer-worker-num` The worker num of the detokenizer manager. `1` Type: int
`--prefill-only-disable-kv-cache` Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa\_skip\_kv\_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page\_size, head\_num, head\_dim) placeholders so GPU memory is not wasted. `False` bool flag (set to enable)
`--model-config-parser` Which model-config parser to use. "auto" picks "mistral" via the is\_mistral\_model name heuristic, else "hf" (AutoConfig over config.json). Plugins can register additional parsers via @register\_model\_config\_parser. `auto` Type: str
## HTTP server
Argument Description Defaults Options
`--host` The host of the HTTP server. `127.0.0.1` Type: str
`--port` The port of the HTTP server. `30000` Type: int
`--fastapi-root-path` App is behind a path based routing proxy. `""` Type: str
`--grpc-mode` If set, use gRPC server instead of HTTP server. `False` bool flag (set to enable)
`--skip-server-warmup` If set, skip warmup. `False` bool flag (set to enable)
`--warmups` Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup\_name1,warmup\_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests `None` Type: str
`--nccl-port` The port for NCCL distributed environment setup. Defaults to a random port. `None` Type: int
`--checkpoint-engine-wait-weights-before-ready` If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests. `False` bool flag (set to enable)
`--ssl-keyfile` The file path to the SSL key file. `None` Type: str
`--ssl-certfile` The file path to the SSL certificate file. `None` Type: str
`--ssl-ca-certs` The CA certificates file. `None` Type: str
`--ssl-keyfile-password` The password to decrypt the SSL keyfile. `None` Type: str
`--enable-ssl-refresh` Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile. `False` bool flag (set to enable)
`--enable-http2` Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang\[http2]'. `False` bool flag (set to enable)
`--http2-max-concurrent-streams` Maximum number of concurrent streams advertised on each HTTP/2 connection (1 to 2^32 - 1). Only applies with --enable-http2. `200` Type: int
## Quantization and data type
Argument Description Defaults Options
`--dtype` Data type for model weights and activations. \* "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. \* "half" for FP16. Recommended for AWQ quantization. \* "float16" is the same as "half". \* "bfloat16" for a balance between precision and range. \* "float" is shorthand for FP32 precision. \* "float32" for FP32 precision. `auto` auto, half, float16, bfloat16, float, float32
`--quantization` The quantization method. `None` awq, fp8, mxfp8, gptq, marlin, gptq\_marlin, awq\_marlin, bitsandbytes, gguf, modelopt, modelopt\_fp8, modelopt\_fp4, nvfp4\_online, modelopt\_mixed, petit\_nvfp4, w8a8\_int8, w8a8\_fp8, moe\_wna16, w4afp8, mxfp4, auto-round, compressed-tensors, modelslim, quark, quark\_int4fp8\_moe, quark\_mxfp4, mlx\_q4, mlx\_q8, unquant
`--quantization-param-path` Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. `None` Type: Optional\[str]
`--kv-cache-dtype` Data type for kv cache storage. "auto" will use model data type. "bf16" or "bfloat16" for BF16 KV cache. "fp8\_e5m2" and "fp8\_e4m3" are supported for CUDA 11.8+. "nvfp4" selects the NVFP4 FP4 E2M1 KV cache recipe; "fp4\_mx\_block16" selects the block-size-16 FP4 E2M1 KV cache recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+ `auto` auto, fp8\_e5m2, fp8\_e4m3, bf16, bfloat16, nvfp4, fp4\_mx\_block16
`--enable-fp32-lm-head` If set, the LM head outputs (logits) are in FP32. `False` bool flag (set to enable)
`--modelopt-quant` The ModelOpt quantization configuration. Supported values: 'fp8', 'int4\_awq', 'w4a8\_awq', 'nvfp4', 'nvfp4\_awq'. This requires the NVIDIA Model Optimizer library to be installed: pip install nvidia-modelopt `None` Type: str
`--modelopt-checkpoint-restore-path` Path to restore a previously saved ModelOpt quantized checkpoint. If provided, the quantization process will be skipped and the model will be loaded from this checkpoint. `None` Type: str
`--modelopt-checkpoint-save-path` Path to save the ModelOpt quantized checkpoint after quantization. This allows reusing the quantized model in future runs. `None` Type: str
`--modelopt-export-path` Path to export the quantized model in HuggingFace format after ModelOpt quantization. The exported model can then be used directly with SGLang for inference. If not provided, the model will not be exported. `None` Type: str
`--quantize-and-serve` Quantize the model with ModelOpt and immediately serve it without exporting. This is useful for development and prototyping. For production, it's recommended to use separate quantization and deployment steps. `False` bool flag (set to enable)
`--rl-quant-profile` Path to the FlashRL quantization profile. Required when using --load-format flash\_rl. `None` Type: str
`--enable-quant-communications` Enable INT8 quantization of TP communications (Supported only for NPU for Qwen3 series). `False` bool flag (set to enable)
`--enable-tf32-matmul` Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set\_float32\_matmul\_precision). CUDA only. Automatically enabled for MiniMax-M2 and GLM-4 models. `False` bool flag (set to enable)
## Memory and scheduling
Argument Description Defaults Options
`--mem-fraction-static` The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. When unset, it is computed as `(GPU memory - reserved memory) / GPU memory`, defaulting to `0.88` if GPU memory cannot be detected. `None` Type: float
`--max-running-requests` The maximum number of running requests. `None` Type: int
`--max-queued-requests` The maximum number of queued requests. This option is ignored when using disaggregation-mode. `None` Type: int
`--max-total-tokens` The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. `None` Type: int
`--chunked-prefill-size` The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. `None` Type: int
`--prefill-max-requests` The maximum number of requests in a prefill batch. If not specified, there is no limit. `None` Type: int
`--enable-dynamic-chunking` Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks. `False` bool flag (set to enable)
`--max-prefill-tokens` The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length. `16384` Type: int
`--schedule-policy` The scheduling policy of the requests. `fcfs` lpm, random, fcfs, dfs-weight, lof, priority, routing-key
`--enable-priority-scheduling` Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. `False` bool flag (set to enable)
`--abort-on-priority-when-disabled` If set, abort requests that specify a priority when priority scheduling is disabled. `False` bool flag (set to enable)
`--schedule-low-priority-values-first` If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. `False` bool flag (set to enable)
`--priority-scheduling-preemption-threshold` Minimum difference in priorities for an incoming request to have to preempt running request(s). `10` Type: int
`--schedule-conservativeness` How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. `1.0` Type: float
`--page-size` The number of tokens in a page. `1` Type: int
`--enable-page-major-kv-layout` Enable the page-major KV layout: lay out the Mamba state and full/SWA KV caches in a page-granularity envelope (page is the outermost axis, layer-major within a page) instead of the default per-layer (layer-major) layout. Requires the Triton attention / linear-attn / Mamba backends (`--attention-backend triton`, and for hybrid models `--linear-attn-backend triton --mamba-backend triton`). `False` bool flag (set to enable)
`--enable-unified-memory` For hybrid Mamba/GDN and hybrid SWA models, replace the statically-partitioned pools (full-attention KV + SWA/Mamba conv/SSM state) with a single byte buffer split dynamically between the sub-pools, so KV-vs-state capacity flexes with the workload instead of being fixed at startup. Implies `--enable-page-major-kv-layout`. Requires the Triton attention / linear-attn / Mamba backends; monolithic (decode) cuda-graph capture only; not yet compatible with PD disaggregation or speculative decoding. `False` bool flag (set to enable)
`--swa-full-tokens-ratio` The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. `0.8` Type: float
`--disable-hybrid-swa-memory` Disable the hybrid SWA memory. `False` bool flag (set to enable)
`--radix-eviction-policy` The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used. `lru` lru, lfu, slru, priority
`--enable-prefill-delayer` Enable prefill delayer for DP attention to reduce idle time. `False` bool flag (set to enable)
`--prefill-delayer-max-delay-passes` Maximum forward passes to delay prefill. `30` Type: int
`--prefill-delayer-token-usage-low-watermark` Token usage low watermark for prefill delayer. `None` Type: float
`--prefill-delayer-queue-min-ratio` Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches `min(running_req * ratio, max_prefill_bs)` so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: `0.1`–`0.5`. `None` Type: float
`--prefill-delayer-max-delay-ms` Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when `--prefill-delayer-queue-min-ratio` is set. Typical: `1000`–`5000`. `None` Type: float
`--prefill-delayer-forward-passes-buckets` Custom buckets for prefill delayer forward passes histogram. 0 and max\_delay\_passes-1 will be auto-added. `None` List\[float]
`--prefill-delayer-wait-seconds-buckets` Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added. `None` List\[float]
`--disable-priority-preemption` Disable priority scheduling preemption. `False` bool flag (set to enable)
`--default-priority-value` Default priority for requests without explicit priority. `None` Type: int
## Runtime options
Argument Description Defaults Options
`--device` The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. `None` Type: str
`--tensor-parallel-size`
`--tp-size`		 The tensor parallelism size. `1` Type: int
`--pipeline-parallel-size`
`--pp-size`		 The pipeline parallelism size. `1` Type: int
`--attention-context-parallel-size`
`--attn-cp-size`		 The attention context parallelism size. `1` Type: int
`--dcp-size`
`--decode-context-parallel-size`		 The decode context parallelism size. See Decode Context Parallelism. `1` Type: int
`--dcp-comm-backend` Communication backend for the DCP attention reduction: AllGather + ReduceScatter, fused NCCL All-to-All, or FlashInfer MNNVL All-to-All. `ag_rs` ag\_rs, a2a, fi\_a2a
`--dcp-replicate-q-proj` For MLA DCP with the a2a/fi\_a2a backend: replicate the Q projection so each DCP rank computes the full-head query locally. Use --no-dcp-replicate-q-proj to disable the model-specific default. None bool flag (set to enable)
`--moe-data-parallel-size`
`--moe-dp-size`		 The moe data parallelism size. `1` Type: int
`--pp-max-micro-batch-size` The maximum micro batch size in pipeline parallelism. None Type: int
`--pp-async-batch-depth` The async batch depth of pipeline parallelism. `0` Type: int
`--stream-interval` The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher `1` Type: int
`--incremental-streaming-output` Whether to output as a sequence of disjoint segments. `False` bool flag (set to enable)
`--random-seed` The random seed. None Type: int
`--mlx-enable-sampling` MLX backend only: sample decode tokens (temperature / top-k / top-p / min-p) instead of greedy argmax. Sampling runs inside the lazy MLX graph, so it works with the overlap scheduler; first tokens from prefill/extend are sampled too. Greedy requests keep exact argmax behavior. Also enables on the MLX path: grammar vocab masks and custom logit processors (these break decode chaining per step; custom processors run on pure-decode steps only), logit\_bias, output logprobs (sampled token / top-k / token\_ids; prompt input logprobs are not computed), NaN sanitization (SGLANG\_SANITIZE\_NAN\_LOGITS), and per-request sampling\_seed under --enable-deterministic-inference (deterministic within MLX only). Penalties are not applied. `False` bool flag (set to enable)
`--constrained-json-whitespace-pattern` (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to \[\n\t ]\* None Type: str
`--constrained-json-disable-any-whitespace` (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. `False` bool flag (set to enable)
`--watchdog-timeout` Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. `300` Type: float
`--soft-watchdog-timeout` Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. `None` Type: float
`--dist-timeout` Set timeout for torch.distributed initialization. `None` Type: int
`--download-dir` Model download directory for huggingface. None Type: str
`--model-checksum` Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID. None Type: str
`--base-gpu-id` The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. `0` Type: int
`--gpu-id-step` The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... `1` Type: int
`--sleep-on-idle` Reduce CPU usage when sglang is idle. `False` bool flag (set to enable)
`--custom-sigquit-handler` Register a custom sigquit handler so you can do additional cleanup after the server is shutdown. This is only available for Engine, not for CLI. `None` Type: str
`--batch-notify-size` Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency. `16` Type: int
`--stream-response-default-include-usage` Include usage in every streaming response (even when stream\_options is not specified). `False` bool flag (set to enable)
`--stream-output` \[Deprecated] Use --incremental-streaming-output instead. — Type: str
`--enable-streaming-session` Enable streaming session mode and StreamingSession wrapper. `False` bool flag (set to enable)
`--load-snapshot-publish-interval` Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately. `15` Type: int
`--use-ray` Use Ray actors for scheduler process management. `False` bool flag (set to enable)
## Logging
Argument Description Defaults Options
`--log-level` The logging level of all loggers. `info` Type: str
`--log-level-http` The logging level of HTTP server. If not set, reuse --log-level by default. `None` Type: str
`--log-requests` Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level `False` bool flag (set to enable)
`--log-requests-level` 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. `2` 0, 1, 2, 3
`--log-requests-format` Format for request logging: 'text' (human-readable) or 'json' (structured) `text` text, json
`--log-requests-target` Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'. `None` List\[str]
`--uvicorn-access-log-exclude-prefixes` Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). `[]` List\[str]
`--crash-dump-folder` Folder for crash diagnostics. Stores completed requests retained by the crash-dump buffer plus in-flight requests and, on NVIDIA CUDA, configures device coredumps under this folder. Existing CUDA coredump environment variables take precedence. If not specified, this option does not enable request crash dumping or set CUDA coredump defaults. `None` Type: str
`--show-time-cost` Show time cost of custom marks. `False` bool flag (set to enable)
`--enable-metrics` Enable log prometheus metrics. `False` bool flag (set to enable)
`--enable-mfu-metrics` Enable estimated MFU-related prometheus metrics. `False` bool flag (set to enable)
`--enable-metrics-for-all-schedulers` Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp\_attention is enabled, as otherwise all metrics appear to come from TP 0. `False` bool flag (set to enable)
`--tokenizer-metrics-custom-labels-header` Specify the HTTP header for passing custom labels for tokenizer metrics. `x-custom-labels` Type: str
`--tokenizer-metrics-allowed-custom-labels` The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., \{'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. `None` List\[str]
`--bucket-time-to-first-token` The buckets of time to first token, specified as a list of floats. `None` List\[float]
`--bucket-inter-token-latency` The buckets of inter-token latency, specified as a list of floats. `None` List\[float]
`--bucket-e2e-request-latency` The buckets of end-to-end request latency, specified as a list of floats. None List\[float]
`--collect-tokens-histogram` Collect prompt/generation tokens histogram. False bool flag (set to enable)
`--prompt-tokens-buckets` The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse \ \ \' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \ \ ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). `None` List\[str]
`--generation-tokens-buckets` The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse \ \ \' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \ \ ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). None List\[str]
`--gc-warning-threshold-secs` The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. `0.0` Type: float
`--decode-log-interval` The log interval of decode batch. `40` Type: int
`--enable-request-time-stats-logging` Enable per request time stats logging `False` bool flag (set to enable)
`--kv-events-config` Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. None Type: str
`--enable-trace` Enable opentelemetry trace `False` bool flag (set to enable)
`--trace-modules` Select the components to trace. Available options are 'request' and 'mooncake'. Format: \,\,...... `request` Type: str
`--otlp-traces-endpoint` Config opentelemetry collector endpoint if --enable-trace is set. format: \:\ `localhost:4317` Type: str
`--grpc-http-sidecar-port` Port for the HTTP sidecar server in gRPC mode (--grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode. `None` Type: int
`--extra-metric-labels` The custom labels for metrics. e.g. '\{"label1": "value1", "label2": "value2"}' `None` Type: str
`--enable-forward-pass-metrics` Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server\_args.forward\_pass\_metrics\_ipc\_name. `False` bool flag (set to enable)
`--forward-pass-metrics-worker-id` — `""` Type: str
`--forward-pass-metrics-ipc-name` — `None` Type: str
## RequestMetricsExporter configuration
Argument Description Defaults Options
`--export-metrics-to-file` Export performance metrics for each request to local file (e.g. for forwarding to external systems). `False` bool flag (set to enable)
`--export-metrics-to-file-dir` Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled). `None` Type: str
## API related
Argument Description Defaults Options
`--api-key` Set API key of the server. It is also used in the OpenAI API compatible server. `None` Type: str
`--admin-api-key` Set admin API key for administrative/control endpoints (e.g., weights update, cache flush, /server\_info). Endpoints marked as admin-only require Authorization: Bearer \ when this is set. `None` Type: str
`--served-model-name` Override the model name returned by the v1/models endpoint in OpenAI API server. `None` Type: str
`--weight-version` Version identifier for the model weights. Defaults to 'default' if not specified. `default` Type: str
`--chat-template` The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. `None` Type: str
`--hf-chat-template-name` When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool\_use', 'rag'), specify which named template to use. If not set, the first available template is used. `None` Type: str
`--completion-template` The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. `None` Type: str
`--file-storage-path` The path of the file storage in backend. `sglang_storage` Type: str
`--enable-cache-report` Return number of cached tokens in usage.prompt\_tokens\_details for each openai request. `False` bool flag (set to enable)
`--reasoning-parser` Specify the parser for reasoning models. Supported parsers: \[deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. `None` deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3
`--tool-call-parser` Specify the parser for handling tool-call interactions. Supported parsers: \[deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi\_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3\_coder, step3]. `None` deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi\_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3\_coder, step3, gigachat3
`--tool-server` Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no external tool server will be used. Native GPT-OSS `web_search` can still be enabled with `EXA_API_KEY`. `None` Type: str
`--sampling-defaults` Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top\_p=1.0, etc.). 'model' uses the model's generation\_config.json to get the recommended sampling parameters if available. Default is 'model'. `model` openai, model
`--strip-thinking-cache` Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents. `False` bool flag (set to enable)
`--enable-strict-thinking` Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering. `False` bool flag (set to enable)
`--asr-max-buffer-seconds` Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer\_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s. `60` Type: int
`--asr-max-concurrent-sessions` Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error\{code:too\_many\_sessions} frame, and closed. Default 32. `32` Type: int
## Data parallelism
Argument Description Defaults Options
`--data-parallel-size`
`--dp-size`		 The data parallelism size. `1` Type: int
`--load-balance-method` The load balancing strategy for data parallelism. The `total_tokens` algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. `auto` auto, round\_robin, follow\_bootstrap\_room, total\_requests, total\_tokens
## Multi-node distributed serving
Argument Description Defaults Options
`--dist-init-addr`
`--nccl-init-addr`		 The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). ` None` Type: str
`--nnodes` The number of nodes. `1` Type: int
`--node-rank` The node rank. `0` Type: int
## Model override args
Argument Description Defaults Options
`--json-model-override-args` A dictionary in JSON string format used to override default model configurations. `{}` Type: str
`--preferred-sampling-params` json-formatted sampling settings that will be returned in /get\_model\_info `None` Type: str
## LoRA
Argument Description Defaults Options
`--enable-lora` Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility. `False` Bool flag (set to enable)
`--enable-lora-overlap-loading` Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. `False` Bool flag (set to enable)
`--max-lora-rank` The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. `None` Type: int
`--lora-target-modules` The union set of all target modules where LoRA should be applied (e.g., q\_proj, k\_proj, gate\_proj). If not specified, it will be automatically inferred from the adapters provided in --lora-paths. You can also set it to all to enable LoRA for all supported modules; note this may introduce minor performance overhead. `None` q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, up\_proj, down\_proj, qkv\_proj, gate\_up\_proj, all
`--lora-paths` The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: \ | \=\ | JSON with schema \{"lora\_name": str, "lora\_path": str, "pinned": bool}. `None` Type: List\[str] / JSON objects
`--max-loras-per-batch` Maximum number of adapters for a running batch, including base-only requests. `8` Type: int
`--max-loaded-loras` If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ --max-loras-per-batch. `None` Type: int
`--lora-eviction-policy` LoRA adapter eviction policy when the GPU memory pool is full. `lru` lru, fifo
`--lora-backend` Choose the kernel backend for multi-LoRA serving. `csgmv` triton, csgmv, ascend, torch\_native
`--max-lora-chunk-size` Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is csgmv. Larger values may improve performance. `16` 16, 32, 64, 128
`--lora-drain-wait-threshold` When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default). `0.0` Type: float
`--experts-shared-outer-loras` Force shared outer LoRA mode for MoE models. When set, w1/w3 lora\_A and w2 lora\_B are shared across experts (expert\_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights. `None` Type: str
`--lora-use-virtual-experts` Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation. `False` bool flag (set to enable)
`--lora-strict-loading` Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error. `False` Type: str
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
Argument Description Defaults Options
`--attention-backend` Choose the kernels for attention layers. `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, cutlass\_mla, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, dual\_chunk\_flash\_attn, aiter, wave, intel\_amx, ascend, intel\_xpu
`--prefill-attention-backend` Choose the kernels for prefill attention layers (have priority over --attention-backend). `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, cutlass\_mla, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, dual\_chunk\_flash\_attn, aiter, wave, intel\_amx, ascend, intel\_xpu
`--decode-attention-backend` Choose the kernels for decode attention layers (have priority over --attention-backend). `None` triton, torch\_native, flex\_attention, dsa, nsa, dsv4, compressed, cutlass\_mla, fa3, fa4, flashinfer, flashmla, trtllm\_mla, cutedsl\_mla, tokenspeed\_mla, trtllm\_mha, dual\_chunk\_flash\_attn, aiter, wave, intel\_amx, ascend, intel\_xpu
`--sampling-backend` Choose the kernels for sampling layers. `None` flashinfer, pytorch, ascend
`--grammar-backend` Choose the backend for grammar-guided decoding. `None` xgrammar, outlines, llguidance, none
`--mm-attention-backend` Set multimodal attention backend. `None` sdpa, fa3, fa4, triton\_attn, ascend\_attn, aiter\_attn, flashinfer\_cudnn, amx\_attn, xpu\_attn
`--dsa-prefill-backend` DSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek DSA-style attention). Auto (hardware-dependent) flashmla\_sparse, flashmla\_sparse\_q8, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
`--dsa-decode-backend` DSA backend for the decode stage when running DeepSeek DSA-style attention. Overrides `--attention-backend` for decoding. Auto (hardware-dependent) flashmla\_sparse, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
`--dsa-topk-backend` Choose the DSA indexer top-k backend. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`. `sgl-kernel` sgl-kernel, torch, flashinfer
`--enable-deepseek-v4-fp4-indexer` Enable the experimental FP4 C4 indexer path for DeepSeek V4. When unset, SGLang keeps the default DeepSeek V4 indexer path. Requires SM100 GPUs with DeepGEMM FP4 indexer support. `False` bool flag (set to enable)
`--fp8-gemm-backend` Choose the runner backend for Blockwise FP8 GEMM operations. For MXFP8 dense GEMM, auto selects flashinfer\_cutedsl when FlashInfer reports support (currently SM100/SM103), and otherwise selects flashinfer\_cutlass on supported Blackwell GPUs. Options also include 'deep\_gemm' (JIT-compiled), 'flashinfer\_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL MXFP8 backend; SM100/SM103 only), 'flashinfer\_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), and 'aiter' (ROCm only). `auto` auto, deep\_gemm, flashinfer\_trtllm, flashinfer\_cutlass, flashinfer\_cutedsl, flashinfer\_deepgemm, cutlass, triton, aiter
`--fp4-gemm-backend` Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects flashinfer\_cutedsl on SM100, marlin on SM80-SM90, flashinfer\_cutlass otherwise (including SM120)), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer\_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80-SM90). Requires FlashInfer to be installed. `auto` auto, flashinfer\_cudnn, flashinfer\_cutedsl, flashinfer\_cutlass, flashinfer\_trtllm, marlin
`--bf16-gemm-backend` Choose the backend for unquantized BF16 GEMM operations. Options: auto (default; selects cutedsl on SM10x GPUs, except deterministic inference selects torch; otherwise uses cuBLAS via torch.nn.functional.linear), cutedsl (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the CuTe DSL kernel and cuBLAS), torch (always uses cuBLAS via torch.nn.functional.linear). `auto` auto, cutedsl, torch
`--disable-flashinfer-autotune` Flashinfer autotune is enabled by default. Set this flag to disable the autotune. `False` bool flag (set to enable)
`--flashinfer-autotune-skip-ops` FlashInfer custom-op identifiers to skip during autotuning. See FlashInfer's autotuning documentation. Skipped ops use the heuristic fallback. SGLang temporarily skips mxfp8\_gemm by default due to an IMA. `None` string
`--radix-cache-backend` Name of a radix-cache backend previously registered via register\_radix\_cache\_backend. Omit this flag to use the built-in default cache selection chain. `None` Type: str
`--nsa-prefill-backend` \[Deprecated] Use --dsa-prefill-backend instead. Auto flashmla\_sparse, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
`--nsa-decode-backend` \[Deprecated] Use --dsa-decode-backend instead. Auto flashmla\_sparse, flashmla\_kv, flashmla\_auto, fa3, tilelang, aiter, trtllm
## Speculative decoding
Argument Description Defaults Options
`--speculative-algorithm` Speculative algorithm. `None` `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`
`--speculative-draft-model-path`
`--speculative-draft-model`		 The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. `None` Type: str
`--speculative-draft-model-revision` The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. `None` Type: str
`--speculative-draft-load-format` The format of the draft model weights to load. If not specified, will use the same format as `--load-format`. Use 'dummy' to initialize draft model weights with random values for profiling. `None` auto, pt, safetensors, npcache, dummy, sharded\_state, gguf, bitsandbytes, mistral, layered, flash\_rl, remote, remote\_instance, fastsafetensors, private, runai\_streamer
`--speculative-num-steps` The number of steps sampled from draft model in Speculative Decoding. `None` Type: int
`--speculative-eagle-topk` The number of tokens sampled from the draft model in eagle2 each step. `None` Type: int
`--speculative-num-draft-tokens` The number of tokens sampled from the draft model in Speculative Decoding. `None` Type: int
`--speculative-accept-threshold-single` Accept a draft token if its probability in the target model is greater than this threshold. `1.0` Type: float
`--speculative-accept-threshold-acc` The accept probability of a draft token is raised from its target probability p to min(1, p / threshold\_acc). `1.0` Type: float
`--speculative-token-map` The path of the draft model's small vocab table. `None` Type: str
`--speculative-attention-mode` Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'. `prefill` prefill, decode
`--speculative-draft-attention-backend` Attention backend for speculative decoding drafting. `None` Same as attention backend options
`--speculative-moe-runner-backend` MOE backend for EAGLE speculative decoding, see `--moe-runner-backend` for options. Same as moe runner backend if unset. `None` auto, deep\_gemm, triton, triton\_kernel, flashinfer\_trtllm, experimental\_sgl\_trtllm, flashinfer\_trtllm\_routed, flashinfer\_cutlass, flashinfer\_mxfp4, flashinfer\_cutedsl, cutlass, aiter, marlin
`--speculative-moe-a2a-backend` MOE A2A backend for EAGLE speculative decoding, see `--moe-a2a-backend` for options. Same as moe a2a backend if unset. `None` none, deepep, mooncake, nixl, mori, ascend\_fuseep, flashinfer, megamoe, pplx
`--speculative-draft-model-quantization` The quantization method for speculative model. `None` Same as `--quantization` options
`--speculative-dflash-block-size` DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH. `None` Type: int
`--speculative-draft-window-size` Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context. `None` Type: int
`--speculative-dflash-draft-window-size` — — Type: int
## Ngram speculative decoding
Argument Description Defaults Options
`--speculative-ngram-min-bfs-breadth` The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. `1` Type: int
`--speculative-ngram-max-bfs-breadth` The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. `10` Type: int
`--speculative-ngram-match-type` Ngram tree-building mode. BFS selects recency-based expansion and PROB selects frequency-based expansion. This setting is forwarded to the ngram cache implementation. `BFS` BFS, PROB
`--speculative-ngram-max-trie-depth` Maximum suffix length stored and matched by the ngram trie. `18` Type: int
`--speculative-ngram-capacity` The cache capacity for ngram speculative decoding. 10000000 Type: int
`--speculative-ngram-external-corpus-path` Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add\_external\_corpus. `None` Type: str
`--speculative-ngram-external-sam-budget` Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding. `0` Type: int
`--speculative-ngram-external-corpus-max-tokens` Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget. `10000000` Type: int
`--speculative-adaptive` Enable adaptive speculative decoding that dynamically adjusts num\_steps based on acceptance rate. `False` bool flag (set to enable)
`--speculative-adaptive-config` Path to a JSON config file for adaptive speculative decoding tuning knobs. `None` Type: str
`--speculative-skip-dp-mlp-sync` Skip the extra MLP sync that the scheduler performs before merging a new batch when speculative decoding + DP attention are both enabled. `False` bool flag (set to enable)
## Multi-layer Eagle speculative decoding
Argument Description Defaults Options
`--enable-multi-layer-eagle` Enable multi-layer Eagle speculative decoding. `False` bool flag (set to enable)
## MoE
Argument Description Defaults Options
`--expert-parallel-size`
`--ep-size`
`--ep`		 The expert parallelism size. `1` Type: int
`--moe-a2a-backend` Select the backend for all-to-all communication for expert parallelism. `none` none, deepep, mooncake, nixl, mori, ascend\_fuseep, flashinfer, megamoe, pplx
`--moe-runner-backend` Choose the runner backend for MoE. `auto` auto, deep\_gemm, triton, triton\_kernel, flashinfer\_trtllm, experimental\_sgl\_trtllm, flashinfer\_trtllm\_routed, flashinfer\_cutlass, flashinfer\_mxfp4, flashinfer\_cutedsl, cutlass, aiter, marlin
`--flashinfer-mxfp4-moe-precision` Choose the computation precision of flashinfer mxfp4 moe `default` default, bf16
`--enable-flashinfer-allreduce-fusion` Enable FlashInfer allreduce fusion with Residual RMSNorm. `False` bool flag (set to enable)
`--enable-aiter-allreduce-fusion` Enable aiter allreduce fusion with Residual RMSNorm. `False` bool flag (set to enable)
`--deepep-mode` Select the mode when enable DeepEP MoE, could be normal, low\_latency or auto. Default is auto, which means low\_latency for decode batch and normal for prefill batch. `auto` normal, low\_latency, auto
`--deepep-dispatcher-output-dtype` Select DeepEP dispather output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 NPU), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input\_global\_scale check → dispatcher\_output\_dtype from quant\_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). `auto` auto, bf16, fp8, int8, nvfp4
`--ep-num-redundant-experts` Allocate this number of redundant experts in expert parallel. `0` Type: int
`--ep-dispatch-algorithm` The algorithm to choose ranks for redundant experts in expert parallel. None Type: str
`--init-expert-location` Initial location of EP experts. `trivial` Type: str
`--enable-eplb` Enable EPLB algorithm `False` bool flag (set to enable)
`--eplb-algorithm` Chosen EPLB algorithm `auto` Type: str
`--eplb-rebalance-num-iterations` Number of iterations to automatically trigger a EPLB re-balance. `1000` Type: int
`--eplb-rebalance-layers-per-chunk` Number of layers to rebalance per forward pass. None Type: int
`--eplb-min-rebalancing-utilization-threshold` Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range \[0.0, 1.0]. `1.0` Type: float
`--expert-distribution-recorder-mode` Mode of expert distribution recorder. ` None` Type: str
`--expert-distribution-recorder-buffer-size` Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. None Type: int
`--expert-balancedness-report-mode` Where to report expert balancedness. off off, server\_log, prometheus, both
`--deepep-config` Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. ` None` Type: str
`--moe-dense-tp-size` TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. ` none` Type: int
`--elastic-ep-backend` Specify the collective communication backend for elastic EP. Currently supports 'mooncake'. ` None` none, mooncake, nixl
`--enable-elastic-expert-backup` Enable elastic EP backend to backup expert weights in DRAM feature. Currently supports 'mooncake'. `False` bool flag (set to enable)
`--mooncake-ib-device` The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled. ` None` Type: str
`--enable-waterfill` Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports `--deepep-mode auto`, `normal`, or `low_latency` when used with DeepEP. Use `auto` or `low_latency` for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set `SGLANG_DISABLE_STATIC_WATERFILL=1` to force dynamic Waterfill with runtime EP all-reduce. `False` bool flag (set to enable)
`--elastic-ep-rejoin` Indicates that this process is a relaunched elastic EP rank that should rejoin an existing process group during rank recovery. `False` bool flag (set to enable)
`--flashinfer-allreduce-fusion-backend` Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. `None` auto, trtllm, mnnvl
`--enforce-disable-flashinfer-allreduce-fusion` Enforce disable FlashInfer allreduce fusion. `False` bool flag (set to enable)
## Mamba Cache
Argument Description Defaults Options
`--max-mamba-cache-size` The maximum size of the mamba cache. `None` Type: int
`--mamba-ssm-dtype` The data type of the SSM states in mamba cache. If not set, read from the model config. Auto (from model config) float32, bfloat16, float16
`--enable-mamba-cache-stochastic-rounding` Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires --mamba-ssm-dtype float16 and CUDA. With --mamba-backend triton, requires SM100. `False` Type: bool
`--mamba-cache-philox-rounds` Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0. `0` Type: int
`--mamba-full-memory-ratio` The ratio of mamba state memory to full kv cache memory. `0.9` Type: float
`--mamba-radix-cache-strategy` The strategy to use for mamba scheduler. auto currently defaults to no\_buffer. 1. no\_buffer does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. extra\_buffer supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes 2x for non-spec; 1+(1/(2+speculative\_num\_draft\_tokens))x for spec dec (e.g. 1.16x if speculative\_num\_draft\_tokens==4)). 2a. extra\_buffer is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra\_buffer supports branching. 3. extra\_buffer\_lazy lowers extra\_buffer's slot cost by allocating one track slot per request instead of two; the second slot is allocated on demand at track-interval boundaries (for speculative decoding it is reserved ahead of each verify window and committed only for accepted boundary crossings). Compatible with speculative decoding (EAGLE/NGRAM/DSPARK/DFLASH); not supported under PD disaggregation. `auto` auto, no\_buffer, extra\_buffer, extra\_buffer\_lazy
`--mamba-track-interval` The interval (in tokens) to track the mamba state during decode. Only used when --mamba-radix-cache-strategy is extra\_buffer. Must be divisible by page\_size if set, and must be >= speculative\_num\_draft\_tokens when using speculative decoding. `256` Type: int
`--enable-int8-mamba-checkpoint` Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for \~2x cached-prefix capacity at fixed memory. `False` bool flag (set to enable)
`--int8-mamba-ckpt-size` Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size). `None` Type: int
`--mamba-backend` Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support). `triton` triton, flashinfer
`--linear-attn-backend` The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend. `triton` triton, cutedsl, flashinfer
`--linear-attn-decode-backend` Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend. `None` triton, cutedsl, flashinfer
`--linear-attn-prefill-backend` Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend. `None` triton, cutedsl, flashinfer
## Hierarchical cache
Argument Description Defaults Options
`--enable-hierarchical-cache` Enable hierarchical cache `False` bool flag (set to enable)
`--hicache-ratio` The ratio of the size of host KV cache memory pool to the size of device pool. `2.0` Type: float
`--hicache-size` The size of host KV cache memory pool in gigabytes, which will override the hicache\_ratio if set. `0` Type: int
`--hicache-write-policy` The write policy of hierarchical cache. `write_through` write\_back, write\_through, write\_through\_selective
`--hicache-io-backend` The IO backend for KV cache transfer between CPU and GPU `kernel` direct, kernel, kernel\_ascend
`--hicache-mem-layout` The layout of host memory pool for hierarchical cache. `page_first` layer\_first, page\_first, page\_first\_direct, page\_first\_kv\_split, page\_head
`--hicache-storage-backend` The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend\_name (custom name), module\_path (Python module path), class\_name (backend class name). `None` file, mooncake, hf3fs, nixl, aibrix, dynamic, eic, simm
`--hicache-storage-prefetch-policy` Control when prefetching from the storage backend should stop. `timeout` best\_effort, wait\_complete, timeout
`--hicache-storage-backend-extra-config` A dictionary in JSON string format, or a string starting with a `@` followed by a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend. `None` Type: str
## Hierarchical sparse attention
Argument Description Defaults Options
`--hisparse-config`
`--hierarchical-sparse-attention-extra-config`		 A dictionary in JSON string format for hierarchical sparse attention configuration. Required fields: `algorithm` (str), `backend` (str). All other fields are algorithm-specific and passed to the algorithm constructor. `None` Type: str
`--enable-hisparse` Enable hierarchical sparse attention `False` bool flag (set to enable)
## LMCache
Argument Description Defaults Options
`--enable-lmcache` Using LMCache as an alternative hierarchical cache solution `False` bool flag (set to enable)
`--lmcache-config-file` Path to the LMCache YAML configuration file `None` Type: str
## Ktransformers
Argument Description Defaults Options
`--kt-weight-path` \[ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder. `None` Type: str
`--kt-method` \[ktransformers parameter] Quantization formats for CPU execution. `AMXINT4` Type: str
`--kt-cpuinfer` \[ktransformers parameter] The number of CPUInfer threads. `None` Type: int
`--kt-threadpool-count` \[ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA). `2` Type: int
`--kt-num-gpu-experts` \[ktransformers parameter] The number of GPU experts. `None` Type: int
`--kt-max-deferred-experts-per-token` \[ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0. `None` Type: int
## Diffusion LLM
Argument Description Defaults Options
`--dllm-algorithm` The diffusion LLM algorithm, such as LowConfidence. `None` Type: str
`--dllm-algorithm-config` The diffusion LLM algorithm configurations. Must be a YAML file. `None` Type: str
`--dllm-fdfo` First-Done-First-Out (FDFO) scheduling lets completed requests leave the batch immediately instead of waiting for slower requests, eliminating head-of-line blocking. Enabled by default; pass `--no-dllm-fdfo` to fall back to synchronous lockstep scheduling. Works with any dLLM algorithm. `True` Type: bool
## Offloading
Argument Description Defaults Options
`--cpu-offload-gb` How many GBs of RAM to reserve for CPU offloading. `0` Type: int
`--offload-group-size` Number of layers per group in offloading. `-1` Type: int
`--offload-num-in-group` Number of layers to be offloaded within a group. `1` Type: int
`--offload-prefetch-step` Steps to prefetch in offloading. `1` Type: int
`--offload-mode` Mode of offloading. `cpu` Type: str
## Args for multi-item scoring
Argument Description Defaults Options
`--enable-mis` Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill. `False` bool flag (set to enable)
## Optimization/debug options
Argument Description Defaults Options
`--disable-radix-cache` Disable RadixAttention for prefix caching. `False` bool flag (set to enable)
`--cuda-graph-config` Canonical per-phase CUDA graph settings as JSON, e.g. . JSON wins over the per-phase --cuda-graph-\* convenience flags and over the legacy flags. Allowed backends: full, breakable, tc\_piecewise, disabled (full is decode-only). `None` Type: JSON (dict-of-dicts)
`--cuda-graph-backend-decode` Backend for the decode phase. Folds into cuda\_graph\_config\[decode].backend. `None` full, breakable, tc\_piecewise, disabled
`--cuda-graph-backend-prefill` Backend for the prefill phase. Folds into cuda\_graph\_config\[prefill].backend. `None` breakable, tc\_piecewise, disabled
`--cuda-graph-max-bs-decode` Maximum batch size captured for the decode CUDA graph. `None` Type: int
`--cuda-graph-max-bs-prefill` Maximum batch size captured for the prefill CUDA graph. `None` Type: int
`--cuda-graph-bs-decode` Explicit list of batch sizes to capture for the decode CUDA graph. `None` List\[int]
`--cuda-graph-bs-prefill` Explicit list of batch sizes to capture for the prefill CUDA graph. `None` List\[int]
`--cuda-graph-tc-compiler` Compiler used by the tc\_piecewise backend (only the prefill phase consumes it today). `None` eager, inductor
`--disable-cuda-graph-padding` Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. `False` bool flag (set to enable)
`--enable-profile-cuda-graph` Enable profiling of cuda graph capture. `False` bool flag (set to enable)
`--debug-cuda-graph` Eager-mode CUDA graph via the breakable backend: graph breaks let every op run eagerly while still going through the capture/replay path. Useful for debugging capture/replay issues. `False` bool flag (set to enable)
`--enable-cudagraph-gc` Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process. `False` bool flag (set to enable)
`--enable-layerwise-nvtx-marker` Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. `False` bool flag (set to enable)
`--enable-nccl-nvls` Enable NCCL NVLS for prefill heavy requests when available. `False` bool flag (set to enable)
`--enable-symm-mem` Enable NCCL symmetric memory for fast collectives. `False` bool flag (set to enable)
`--disable-flashinfer-cutlass-moe-fp4-allgather` Disables quantize before all-gather for flashinfer cutlass moe. `False` bool flag (set to enable)
`--enable-tokenizer-batch-encode` Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input\_ids, or input\_embeds. `False` bool flag (set to enable)
`--disable-tokenizer-batch-decode` Disable batch decoding when decoding multiple completions. `False` bool flag (set to enable)
`--disable-outlines-disk-cache` Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency. `False` bool flag (set to enable)
`--disable-custom-all-reduce` Disable the custom all-reduce kernel and fall back to NCCL. `False` bool flag (set to enable)
`--enable-mscclpp` Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. `False` bool flag (set to enable)
`--enable-torch-symm-mem` Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. `False` bool flag (set to enable)
`--disable-overlap-schedule` Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. `False` bool flag (set to enable)
`--enable-mixed-chunk` Enabling mixing prefill and decode in a batch when using chunked prefill. `False` bool flag (set to enable)
`--enable-dp-attention` Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported. `False` bool flag (set to enable)
`--enable-dp-lm-head` Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. `False` bool flag (set to enable)
`--enable-two-batch-overlap` Enabling two micro batches to overlap. `False` bool flag (set to enable)
`--enable-single-batch-overlap` Let computation and communication overlap within one micro batch. `False` bool flag (set to enable)
`--tbo-token-distribution-threshold` The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. `0.48` Type: float
`--enable-torch-compile` Optimize the model with torch.compile. Experimental feature. `False` bool flag (set to enable)
`--enable-torch-compile-debug-mode` Enable debug mode for torch compile. `False` bool flag (set to enable)
`--torch-compile-max-bs` Set the maximum batch size when using torch compile. `32` Type: int
`--cuda-graph-max-bs-decode` Deprecated alias for --cuda-graph-max-bs-decode. `None` Type: int
`--cuda-graph-bs` Deprecated alias for --cuda-graph-bs-decode. `None` List\[int]
`--disable-cuda-graph` Deprecated. Use --cuda-graph-backend-decode=disabled and/or --cuda-graph-backend-prefill=disabled. False bool flag (set to enable)
`--enable-breakable-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=breakable. False bool flag (set to enable)
`--disable-prefill-cuda-graph` Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled. `False` bool flag (set to enable)
`--disable-decode-cuda-graph` Disable the decode-phase CUDA graph. Convenience for --cuda-graph-backend-decode=disabled. `False` bool flag (set to enable)
`--disable-piecewise-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=disabled. False bool flag (set to enable)
`--enforce-piecewise-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=tc\_piecewise. Explicitly setting the prefill backend now skips the auto-disable cascade automatically. False bool flag (set to enable)
`--piecewise-cuda-graph-tokens` Deprecated alias for --cuda-graph-bs-prefill. `None` List\[int]
`--piecewise-cuda-graph-compiler` Deprecated alias for --cuda-graph-tc-compiler. eager eager, inductor
`--piecewise-cuda-graph-max-tokens` Deprecated alias for --cuda-graph-max-bs-prefill. 4096 Type: int
`--enable-p2p-check` Enable P2P check for GPU access, otherwise the p2p access is allowed by default. `False` bool flag (set to enable)
`--triton-attention-reduce-in-fp32` Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels. `False` bool flag (set to enable)
`--triton-attention-num-kv-splits` The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. `8` Type: int
`--triton-attention-split-tile-size` The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. `None` Type: int
`--num-continuous-decode-steps` Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. `1` Type: int
`--delete-ckpt-after-loading` Delete the model checkpoint after loading the model. `False` bool flag (set to enable)
`--enable-memory-saver` Allow saving memory using release\_memory\_occupation and resume\_memory\_occupation `False` bool flag (set to enable)
`--enable-weights-cpu-backup` Save model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation `False` bool flag (set to enable)
`--enable-draft-weights-cpu-backup` Save draft model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation `False` bool flag (set to enable)
`--allow-auto-truncate` Allow automatically truncating requests that exceed the maximum input length instead of returning an error. `False` bool flag (set to enable)
`--enable-custom-logit-processor` Enable users to pass custom logit processors to the server (disabled by default for security) `False` bool flag (set to enable)
`--flashinfer-mla-disable-ragged` Not using ragged prefill wrapper when running flashinfer mla `False` bool flag (set to enable)
`--disable-shared-experts-fusion` Disable shared experts fusion optimization for deepseek v3/r1. `False` bool flag (set to enable)
`--disable-chunked-prefix-cache` Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. `False` bool flag (set to enable)
`--image-processor-backend` Image processor backend. `auto` lets Transformers select the best available backend. `auto` `auto`, `torchvision`, `pil`
`--disable-fast-image-processor` Deprecated. Use `--image-processor-backend=pil` instead. `False` bool flag (set to enable)
`--keep-mm-feature-on-device` Keep multimodal feature tensors on device after processing to save D2H copy. `False` bool flag (set to enable)
`--enable-return-hidden-states` Enable returning hidden states with responses. `False` bool flag (set to enable)
`--enable-return-routed-experts` Enable returning routed experts of each layer with responses. `False` bool flag (set to enable)
`--scheduler-recv-interval` The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. `1` Type: int
`--numa-node` Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. `None` List\[int]
`--enable-deterministic-inference` Enable deterministic inference mode with batch invariant ops. `False` bool flag (set to enable)
`--rl-on-policy-target` The training system that SGLang needs to match for true on-policy. `None` fsdp
`--enable-attn-tp-input-scattered` Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. `False` bool flag (set to enable)
`--enable-prefill-cp` Enable context parallelism for the prefill phase. Select the layout with --cp-strategy. `False` bool flag (set to enable)
`--cp-strategy` Sharding strategy for prefill CP. zigzag is the former in-seq-split mode; interleave is the former round-robin-split mode. None zigzag, interleave
`--enable-fused-qk-norm-rope` Enable fused qk normalization and rope rotary embedding. `False` bool flag (set to enable)
`--enable-precise-embedding-interpolation` Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values. `False` bool flag (set to enable)
`--kv-canary` KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane). `none` none, log, raise
`--kv-canary-real-data` Check the real KV-cache in the canary. 'none' (default) disables the feature. 'partial' checks the first 16 bytes of each real-KV slot. 'all' checks the full real-KV slot. `none` Type: str
`--kv-canary-sweep-interval` Every N forward steps, run a full-pool sweep. `0` Type: int
`--pre-warm-nccl` Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL). `False` bool flag (set to enable)
`--enable-dp-attention-local-control-broadcast` With DP-attention, send control messages to every DP group leader and broadcast within attn\_tp\_group instead of the full tp\_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration. `False` bool flag (set to enable)
`--enforce-shared-experts-fusion` Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion. `False` bool flag (set to enable)
`--enable-return-indexer-topk` Enable returning indexer topk indices of layers with indexer with responses. `False` bool flag (set to enable)
`--disable-attn-tp-gather` Disable scheduler-side attn\_tp\_gather (the upstream SP path that pads num\_tokens to attn\_tp\_size and pre-allocates a gathered buffer). Use for models that manage SP scatter/gather at the model level (e.g., perform their own all\_gather/reduce\_scatter inside attention) and do not consume the upstream gathered\_buffer. Without this, the cuda graph runner pads num\_tokens to attn\_tp\_size, which can cause kernel autotuners to select wrong-sized variants at small batches. `False` bool flag (set to enable)
`--enable-dsa-prefill-context-parallel` \[Deprecated] Use --enable-prefill-cp instead. — Type: str
`--enable-nsa-prefill-context-parallel` \[Deprecated] Use --enable-prefill-cp instead. — Type: str
`--enable-prefill-context-parallel` \[Deprecated] Use --enable-prefill-cp instead. — Type: str
`--dsa-prefill-cp-mode` \[Deprecated] Use --cp-strategy \{zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'; 'round-robin-split' maps to 'interleave'. `round-robin-split` in-seq-split, round-robin-split
`--nsa-prefill-cp-mode` \[Deprecated] Use --cp-strategy instead. Auto in-seq-split, round-robin-split
`--prefill-cp-mode` \[Deprecated] Use --cp-strategy \{zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'. `in-seq-split` in-seq-split
`--enable-fused-moe-sum-all-reduce` Enable fused moe triton and sum all reduce. `False` bool flag (set to enable)
`--gc-threshold` Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers. — Type: int (one or more)
## Dynamic batch tokenizer
Argument Description Defaults Options
`--enable-dynamic-batch-tokenizer` Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. `False` bool flag (set to enable)
`--dynamic-batch-tokenizer-batch-size` \[Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. `32` Type: int
`--dynamic-batch-tokenizer-batch-timeout` \[Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. `0.002` Type: float
## Debug tensor dumps
Argument Description Defaults Options
`--debug-tensor-dump-output-folder` The output folder for dumping tensors. `None` Type: str
`--debug-tensor-dump-layers` The layer ids to dump. Dump all layers if not specified. `None` Type: JSON list
`--debug-tensor-dump-input-file` The input filename for dumping tensors `None` Type: str
`--debug-tensor-dump-inject` Inject the outputs from jax as the input of every layer. `False` Type: str
## PD disaggregation
Argument Description Defaults Options
`--disaggregation-mode` Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated `null` null, prefill, decode
`--disaggregation-transfer-backend` The backend for disaggregation transfer. Default is mooncake. `mooncake` mooncake, nixl, ascend, fake, mori, mooncake\_tcp
`--disaggregation-bootstrap-port` Bootstrap server port on the prefill server. Default is 8998. `8998` Type: int
`--disaggregation-ib-device` The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5\_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. None Type: str
`--disaggregation-decode-enable-offload-kvcache` Enable async KV cache offloading on decode server (PD mode). `False` bool flag (set to enable)
`--num-reserved-decode-tokens` Number of decode tokens that will have memory reserved when adding new request to the running batch. `512` Type: int
`--disaggregation-decode-polling-interval` The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. `1` Type: int
`--disaggregation-decode-enable-radix-cache` Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake. `False` bool flag (set to enable)
`--optimistic-prefill-retries` Number of optimistic prefill retries that will skip the bootstrap wait. `0` Type: int
## Encode prefill disaggregation
Argument Description Defaults Options
`--encoder-only` For MLLM with an encoder, launch an encoder-only server `False` bool flag (set to enable)
`--language-only` For VLM, load weights for the language model only. `False` bool flag (set to enable)
`--encoder-transfer-backend` The backend for encoder disaggregation transfer. Default is zmq\_to\_scheduler. `zmq_to_scheduler` zmq\_to\_scheduler, zmq\_to\_tokenizer, mooncake
`--encoder-urls` List of encoder server urls. `[]` Type: JSON list
`--encoder-bootstrap-port` Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here. `8997` Type: int
`--encoder-register-urls` One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls [http://prefill0:8997](http://prefill0:8997) [http://prefill1:8997](http://prefill1:8997). Used with --encoder-only servers. `[]` Type: str (one or more)
`--enable-adaptive-dispatch-to-encoder` When enabled, adaptively dispatch: multi-image requests go to encoder in language\_only epd mode, single-image requests are processed locally. `False` bool flag (set to enable)
## Custom weight loader
Argument Description Defaults Options
`--custom-weight-loader` The custom dataloader which used to update the model. Should be set with a valid import path, such as my\_package.weight\_load\_func None List\[str]
`--weight-loader-disable-mmap` Disable mmap while loading weight using safetensors. `False` bool flag (set to enable)
`--weight-loader-prefetch-checkpoints` Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards in a background thread, reducing total network I/O on shared filesystems (NFS/Lustre) from N\*checkpoint to 1\*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set `enable_multithread_load=true` in `--model-loader-extra-config` to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op). `False` bool flag (set to enable)
`--weight-loader-prefetch-num-threads` Number of threads per rank for checkpoint prefetching. `4` Type: int
`--remote-instance-weight-loader-seed-instance-ip` The ip of the seed instance for loading weights from remote instance. None Type: str
`--remote-instance-weight-loader-seed-instance-service-port` The service port of the seed instance for loading weights from remote instance. None Type: int
`--remote-instance-weight-loader-send-weights-group-ports` The communication group ports for loading weights from remote instance. None Type: JSON list
`--remote-instance-weight-loader-backend` The backend for loading weights from remote instance. Can be 'transfer\_engine', 'nccl', or 'modelexpress'. Default is 'nccl'. `nccl` transfer\_engine, nccl, modelexpress
`--remote-instance-weight-loader-start-seed-via-transfer-engine` Start seed server via transfer engine backend for remote instance weight loader. `False` bool flag (set to enable)
`--weight-loader-drop-cache-after-load` Call posix\_fadvise(DONTNEED) on each safetensors shard after loading it. `False` bool flag (set to enable)
`--engine-info-bootstrap-port` Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node. `6789` Type: int
`--modelexpress-config` JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer\_engine"). Example: '\{"url": "localhost:8001", "transport": "nixl"}' `None` Type: str
## For PD-Multiplexing
Argument Description Defaults Options
`--enable-pdmux` Enable PD-Multiplexing, PD running on greenctx stream. `False` bool flag (set to enable)
`--pdmux-config-path` The path of the PD-Multiplexing config file. `None` Type: str
`--sm-group-num` Number of sm partition groups. `8` Type: int
## Configuration file support
Argument Description Defaults Options
`--config` Read CLI options from a config file. Must be a YAML file with configuration options. `None` Type: str
## For Multi-Modal
Argument Description Defaults Options
`--enable-broadcast-mm-inputs-process` Enable broadcast mm-inputs process in scheduler. `False` bool flag (set to enable)
`--mm-process-config` Multimodal preprocessing config, a json config contains keys: image, video, audio. \{} Type: JSON / Dict
`--allowed-media-domains` Restrict client-supplied HTTP(S) media URLs and redirect destinations to these exact hostnames. Unrestricted Space-separated hostnames
`--media-url-max-file-size-mb` Maximum streamed size in MiB for one remote media download. Set to 0 to disable the limit. `64` Type: int
`--mm-enable-dp-encoder` Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically. `False` bool flag (set to enable)
`--limit-mm-data-per-request` Limit the number of multimodal inputs per request. e.g. '\{"image": 1, "video": 1, "audio": 1}' `None` Type: JSON / Dict
`--enable-mm-global-cache` Enable Mooncake-backed global multimodal embedding cache on encoder servers so repeated images can reuse cached ViT embeddings instead of recomputing them. `False` bool flag (set to enable)
## For checkpoint decryption
Argument Description Defaults Options
`--decrypted-config-file` The path of the decrypted config file. `None` Type: str
`--decrypted-draft-config-file` The path of the decrypted draft config file. `None` Type: str
`--enable-prefix-mm-cache` Enable prefix multimodal cache. Currently only supports mm-only. `False` bool flag (set to enable)
## Forward hooks
Argument Description Defaults Options
`--forward-hooks` JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory. `None` Type: JSON list
## For MindStudio-probe(msProbe) dump
Argument Description Defaults Options
`--msprobe-dump-config` The path of the JSON configuration file for msProbe. If specified, enables msProbe dump. `None` Type: str
## Deprecated arguments
Argument Description Defaults Options
`--prefill-round-robin-balance` Note: Note: --prefill-round-robin-balance is deprecated now. `None` N/A
`--hybrid-kvcache-ratio` Mix ratio in \[0,1] between uniform and hybrid kv buffers (0.0 = pure uniform: swa\_size / full\_size = 1)(1.0 = pure hybrid: swa\_size / full\_size = local\_attention\_size / context\_length) `None` Optional\[float]
# Session-Aware Radix Cache Source: https://docs.sglang.io/docs/advanced_features/session_radix_cache Session-aware radix caching improves cache hits for long-lived, multi-turn workloads under memory pressure. It registers reusable KV to a session and evicts unreferenced KV before KV still referenced by an active session. Session references are soft protection, not memory pins. Referenced KV can still be evicted when reclaiming unreferenced KV is insufficient. ## Enable the cache This feature is implemented only by `UnifiedRadixCache`. ```bash Command theme={null} SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --enable-session-radix-cache ``` ## Pass and close the session Your application must pass the same top-level `session_id` on every request in a session. The ID labels cache references only; it does not append or reconstruct conversation context, so each request must still contain the intended prompt. ```bash Command theme={null} curl http://localhost:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "FULL_PROMPT_FOR_THIS_TURN", "sampling_params": {"max_new_tokens": 128}, "session_id": "agent-42" }' ``` When a request finishes, SGLang automatically registers its reusable cache leaves under the `session_id`. This reference-only workflow does not require an `/open_session` call. Call `/close_session` when the application session ends, including error and cancellation paths: ```bash Command theme={null} curl -X POST http://localhost:30000/close_session \ -H "Content-Type: application/json" \ -d '{"session_id": "agent-42"}' ``` Closing removes the session's references but does not immediately free its KV. The KV remains reusable and returns to the normal eviction order. ## Eviction behavior The cache tracks references independently for each UnifiedRadixCache component. Device and host eviction use the same session preference. | Component | Referenced data | Eviction order | | ------------------------------ | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Full attention | The reusable prefix path from the registered leaf to the root | Unreferenced nodes first, then referenced nodes with fewer session references, then the configured policy such as LRU | | Sliding-window attention (SWA) | The reusable tail covering the sliding window plus page-alignment allowance | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required | | Mamba | The reusable state on the registered leaf | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required | UnifiedRadixCache still applies component cascade rules. Evicting an internal Full node also evicts its SWA and Mamba data; evicting SWA also evicts Mamba data; evicting Mamba affects only Mamba. Evicting a leaf removes all component data on that leaf. # SGLang Model Gateway Source: https://docs.sglang.io/docs/advanced_features/sgl_model_gateway SGLang Model Gateway is a high-performance model-routing gateway for large-scale LLM deployments. It centralizes worker lifecycle management, balances traffic across heterogeneous protocols (HTTP, gRPC, OpenAI-compatible), and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. The gateway is deeply optimized for the SGLang serving runtime, but can route to any OpenAI-compatible backend. *** ## Table of Contents 1. [Overview](#overview) 2. [Architecture](#architecture) * [Control Plane](#control-plane) * [Data Plane](#data-plane) * [Storage and Privacy](#storage-and-privacy) 3. [Installation](#installation) 4. [Quick Start](#quick-start) 5. [Deployment Modes](#deployment-modes) * [Co-launch Router and Workers](#co-launch-router-and-workers) * [Separate Launch (HTTP)](#separate-launch-http) * [gRPC Launch](#grpc-launch) * [Prefill-Decode Disaggregation](#prefill-decode-disaggregation) * [OpenAI Backend Proxy](#openai-backend-proxy) * [Multi-Model Inference Gateway](#multi-model-inference-gateway) 6. [API Reference](#api-reference) * [Inference Endpoints](#inference-endpoints) * [Tokenization Endpoints](#tokenization-endpoints) * [Parser Endpoints](#parser-endpoints) * [Classification API](#classification-api) * [Conversation and Response APIs](#conversation-and-response-apis) * [Worker Management APIs](#worker-management-apis) * [Admin and Health Endpoints](#admin-and-health-endpoints) 7. [Load Balancing Policies](#load-balancing-policies) 8. [Reliability and Flow Control](#reliability-and-flow-control) * [Retries](#retries) * [Circuit Breaker](#circuit-breaker) * [Rate Limiting and Queuing](#rate-limiting-and-queuing) * [Health Checks](#health-checks) 9. [Reasoning Parser Integration](#reasoning-parser-integration) 10. [Tool Call Parsing](#tool-call-parsing) 11. [Tokenizer Management](#tokenizer-management) 12. [MCP Integration](#mcp-integration) 13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes) 14. [History and Data Connectors](#history-and-data-connectors) 15. [WASM Middleware](#wasm-middleware) 16. [Language Bindings](#language-bindings) 17. [Security and Authentication](#security-and-authentication) * [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server) * [mTLS for Worker Communication](#mtls-for-worker-communication) 18. [Observability](#observability) * [Prometheus Metrics](#prometheus-metrics) * [OpenTelemetry Tracing](#opentelemetry-tracing) * [Logging](#logging) 19. [Production Recommendations](#production-recommendations) * [Security Best Practices](#security-best-practices) * [High Availability](#high-availability) * [Performance](#performance) * [Kubernetes Deployment](#kubernetes-deployment) * [Monitoring with PromQL](#monitoring-with-promql) 20. [Configuration Reference](#configuration-reference) 21. [Troubleshooting](#troubleshooting) *** ## Overview * **Unified control plane** for registering, monitoring, and orchestrating regular, prefill, and decode workers across heterogeneous model fleets. * **Multi-protocol data plane** that routes traffic across HTTP, PD (prefill/decode), gRPC, and OpenAI-compatible backends with shared reliability primitives. * **Industry-first gRPC pipeline** with native Rust tokenization, reasoning parsers, and tool-call execution for high-throughput, OpenAI-compatible serving; supports both single-stage and PD topologies. * **Inference Gateway Mode (`--enable-igw`)** dynamically instantiates multiple router stacks (HTTP regular/PD, gRPC) and applies per-model policies for multi-tenant deployments. * **Conversation & responses connectors** centralize chat history inside the router so the same context can be reused across models and MCP loops without leaking data to upstream vendors (memory, none, Oracle ATP, PostgreSQL). * **Enterprise privacy**: agentic multi-turn `/v1/responses`, native MCP client (STDIO/HTTP/SSE/Streamable), and history storage all operate within the router boundary. * **Reliability core**: retries with jitter, worker-scoped circuit breakers, token-bucket rate limiting with queuing, background health checks, and cache-aware load monitoring. * **Comprehensive observability**: 40+ Prometheus metrics, OpenTelemetry distributed tracing, structured logging, and request ID propagation. *** ## Architecture ### Control Plane * **Worker Manager** discovers capabilities (`/get_server_info`, `/get_model_info`), tracks load, and registers/removes workers in the shared registry. * **Job Queue** serializes add/remove requests and exposes status (`/workers/{worker_id}`) so clients can track onboarding progress. * **Load Monitor** feeds cache-aware and power-of-two policies with live worker load statistics. * **Health Checker** continuously probes workers and updates readiness, circuit breaker state, and router metrics. * **Tokenizer Registry** manages dynamically registered tokenizers with async loading from HuggingFace or local paths. ### Data Plane * **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`, and associated admin endpoints. * **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings and classification. * **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local. ### Storage and Privacy * Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors. * `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads. *** ## Installation ### Docker Pre-built Docker images are available on Docker Hub with multi-architecture support (x86\_64 and ARM64): ```bash Command theme={null} docker pull lmsysorg/sgl-model-gateway:latest ``` ### Prerequisites * **Rust and Cargo** ```bash Command theme={null} curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source "$HOME/.cargo/env" rustc --version cargo --version ``` * **Python** with `pip` and virtualenv tooling available. ### Rust Binary ```bash Command theme={null} cd sgl-model-gateway cargo build --release ``` ### Python Package ```bash Command theme={null} pip install maturin # Fast development mode cd sgl-model-gateway/bindings/python maturin develop # Production build maturin build --release --out dist --features vendored-openssl pip install --force-reinstall dist/*.whl ``` *** ## Quick Start ### Regular HTTP Routing ```bash Command theme={null} # Rust binary ./target/release/sgl-model-gateway \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware # Python launcher python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8000 \ --policy cache_aware ``` ### gRPC Routing ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --reasoning-parser deepseek-r1 \ --tool-call-parser json \ --host 0.0.0.0 --port 8080 ``` *** ## Deployment Modes ### Co-launch Router and Workers Launch the router and a fleet of SGLang workers in one process: ```bash Command theme={null} python -m sglang_router.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --dp-size 4 \ --host 0.0.0.0 \ --port 30000 ``` Comprehensive example with router arguments (prefixed with `--router-`): ```bash Command theme={null} python -m sglang_router.launch_server \ --host 0.0.0.0 \ --port 8080 \ --model meta-llama/Llama-3.1-8B-Instruct \ --tp-size 1 \ --dp-size 8 \ --grpc-mode \ --log-level debug \ --router-prometheus-port 10001 \ --router-tool-call-parser llama \ --router-model-path meta-llama/Llama-3.1-8B-Instruct \ --router-policy round_robin \ --router-log-level debug ``` ### Separate Launch (HTTP) Run workers independently and point the router at their HTTP endpoints: ```bash Command theme={null} # Worker nodes python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000 python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8001 # Router node python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --policy cache_aware \ --host 0.0.0.0 --port 30000 ``` ### gRPC Launch Use SRT gRPC workers to unlock the highest throughput and access native reasoning/tool pipelines: ```bash Command theme={null} # Workers expose gRPC endpoints python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --grpc-mode \ --port 20000 # Router python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --reasoning-parser deepseek-r1 \ --tool-call-parser json \ --host 0.0.0.0 --port 8080 ``` The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC. ### Prefill-Decode Disaggregation Split prefill and decode workers for PD-aware caching and balancing: ```bash Command theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://prefill1:30001 9001 \ --decode http://decode1:30011 \ --prefill-policy cache_aware \ --decode-policy power_of_two ``` Prefill entries accept an optional bootstrap port. PD mode merges prefill metadata with decode outputs and streams results back to the client. ### OpenAI Backend Proxy Proxy OpenAI-compatible endpoints while keeping history and MCP sessions local: ```bash Command theme={null} python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend memory ``` OpenAI backend mode expects exactly one `--worker-urls` entry per router instance. ### Multi-Model Inference Gateway Enable IGW mode to route multiple models through a single router: ```bash Command theme={null} ./target/release/sgl-model-gateway \ --enable-igw \ --policy cache_aware \ --max-concurrent-requests 512 # Register workers dynamically curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{ "url": "http://worker-a:8000", "model_id": "mistral", "priority": 10, "labels": {"tier": "gold"} }' ``` *** ## API Reference ### Inference Endpoints
Method Path Description
`POST` `/generate` SGLang generate API
`POST` `/v1/chat/completions` OpenAI-compatible chat completions (streaming/tool calls)
`POST` `/v1/completions` OpenAI-compatible text completions
`POST` `/v1/embeddings` Embedding generation (HTTP and gRPC)
`POST` `/v1/rerank`, `/rerank` Reranking requests
`POST` `/v1/classify` Text classification
### Tokenization Endpoints The gateway provides HTTP endpoints for text tokenization with batch support, designed to mirror the SGLang Python tokenization API.
Method Path Description
`POST` `/v1/tokenize` Tokenize text to token IDs (single or batch)
`POST` `/v1/detokenize` Convert token IDs back to text (single or batch)
`POST` `/v1/tokenizers` Register a new tokenizer (async, returns job status)
`GET` `/v1/tokenizers` List all registered tokenizers
`GET` `/v1/tokenizers/{id}` Get tokenizer info by UUID
`GET` `/v1/tokenizers/{id}/status` Check async tokenizer loading status
`DELETE` `/v1/tokenizers/{id}` Remove a tokenizer from the registry
#### Tokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "Hello, world!" } ``` #### Batch Tokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": ["Hello", "World", "How are you?"] } ``` #### Tokenize Response ```json Config theme={null} { "tokens": [15339, 11, 1917, 0], "count": 4, "char_count": 13 } ``` #### Detokenize Request ```json Config theme={null} { "model": "meta-llama/Llama-3.1-8B-Instruct", "tokens": [15339, 11, 1917, 0], "skip_special_tokens": true } ``` #### Detokenize Response ```json Config theme={null} { "text": "Hello, world!" } ``` #### Add Tokenizer (Async) ```bash Command theme={null} curl -X POST http://localhost:30000/v1/tokenizers \ -H "Content-Type: application/json" \ -d '{"name": "llama3", "source": "meta-llama/Llama-3.1-8B-Instruct"}' ``` Response: ```json Config theme={null} { "id": "550e8400-e29b-41d4-a716-446655440000", "status": "pending", "message": "Tokenizer registration queued" } ``` Check status: ```bash Command theme={null} curl http://localhost:30000/v1/tokenizers/550e8400-e29b-41d4-a716-446655440000/status ``` ### Parser Endpoints The gateway provides admin endpoints for parsing reasoning content and function calls from LLM outputs.
Method Path Description
`POST` `/parse/reasoning` Separate reasoning (`<think>`) from normal text
`POST` `/parse/function_call` Parse function/tool calls from text
#### Separate Reasoning Request ```json Config theme={null} { "text": "<think>Let me analyze this step by step...</think>The answer is 42.", "parser": "deepseek-r1" } ``` #### Response ```json Config theme={null} { "normal_text": "The answer is 42.", "reasoning_text": "Let me analyze this step by step..." } ``` #### Function Call Parsing ```json Config theme={null} { "text": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}", "parser": "json" } ``` ### Classification API The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`). #### Request ```bash Command theme={null} curl http://localhost:30000/v1/classify \ -H "Content-Type: application/json" \ -d '{ "model": "jason9693/Qwen2.5-1.5B-apeach", "input": "I love this product!" }' ``` #### Response ```json Config theme={null} { "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab", "object": "list", "created": 1767034308, "model": "jason9693/Qwen2.5-1.5B-apeach", "data": [ { "index": 0, "label": "positive", "probs": [0.12, 0.88], "num_classes": 2 } ], "usage": { "prompt_tokens": 6, "completion_tokens": 0, "total_tokens": 6 } } ``` #### Response Fields
Field Description
`label` Predicted class label (from model's `id2label` config, or `LABEL_N` fallback)
`probs` Probability distribution over all classes (softmax of logits)
`num_classes` Number of classification classes
#### Notes * Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax * Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.) * Both HTTP and gRPC routers support classification ### Conversation and Response APIs
Method Path Description
`POST` `/v1/responses` Create background responses (agentic loops)
`GET` `/v1/responses/{id}` Retrieve stored response
`POST` `/v1/responses/{id}/cancel` Cancel background response
`DELETE` `/v1/responses/{id}` Delete response
`GET` `/v1/responses/{id}/input_items` List response input items
`POST` `/v1/conversations` Create conversation
`GET` `/v1/conversations/{id}` Get conversation
`POST` `/v1/conversations/{id}` Update conversation
`DELETE` `/v1/conversations/{id}` Delete conversation
`GET` `/v1/conversations/{id}/items` List conversation items
`POST` `/v1/conversations/{id}/items` Add items to conversation
`GET` `/v1/conversations/{id}/items/{item_id}` Get conversation item
`DELETE` `/v1/conversations/{id}/items/{item_id}` Delete conversation item
### Worker Management APIs
Method Path Description
`POST` `/workers` Queue worker registration (returns 202 Accepted)
`GET` `/workers` List workers with health, load, and policy metadata
`GET` `/workers/{worker_id}` Inspect specific worker or job queue entry
`PUT` `/workers/{worker_id}` Queue worker update
`DELETE` `/workers/{worker_id}` Queue worker removal
#### Add Worker ```bash Command theme={null} curl -X POST http://localhost:30000/workers \ -H "Content-Type: application/json" \ -d '{"url":"grpc://0.0.0.0:31000","worker_type":"regular"}' ``` #### List Workers ```bash Command theme={null} curl http://localhost:30000/workers ``` Response: ```json Config theme={null} { "workers": [ { "id": "2f3a0c3e-3a7b-4c3f-8c70-1b7d4c3a6e1f", "url": "http://0.0.0.0:31378", "model_id": "mistral", "priority": 50, "cost": 1.0, "worker_type": "regular", "is_healthy": true, "load": 0, "connection_mode": "Http" } ], "total": 1, "stats": { "prefill_count": 0, "decode_count": 0, "regular_count": 1 } } ``` ### Admin and Health Endpoints
Method Path Description
`GET` `/liveness` Health check (always returns OK)
`GET` `/readiness` Readiness check (checks healthy worker availability)
`GET` `/health` Alias for liveness
`GET` `/health_generate` Health generate test
`GET` `/engine_metrics` Engine-level metrics from workers
`GET` `/v1/models` List available models
`GET` `/get_model_info` Get model information
`GET` `/get_server_info` Get server information
`POST` `/flush_cache` Clear all caches
`GET` `/get_loads` Get all worker loads
`POST` `/wasm` Upload WASM module
`GET` `/wasm` List WASM modules
`DELETE` `/wasm/{module_uuid}` Remove WASM module
*** ## Load Balancing Policies
Policy Description Usage
`random` Uniform random selection `--policy random`
`round_robin` Cycles through workers in order `--policy round_robin`
`power_of_two` Samples two workers and picks the lighter one `--policy power_of_two`
`cache_aware` Combines cache locality with load balancing (default) `--policy cache_aware`
`bucket` Divides workers into load buckets with dynamic boundaries `--policy bucket`
### Cache-Aware Policy Tuning ```bash Command theme={null} --cache-threshold 0.5 \ --balance-abs-threshold 32 \ --balance-rel-threshold 1.5 \ --eviction-interval-secs 120 \ --max-tree-size 67108864 ```
Parameter Default Description
`--cache-threshold` 0.3 Minimum prefix match ratio for cache hit
`--balance-abs-threshold` 64 Absolute load difference before rebalancing
`--balance-rel-threshold` 1.5 Relative load ratio before rebalancing
`--eviction-interval-secs` 120 Cache eviction cadence in seconds
`--max-tree-size` 67108864 Maximum nodes in cache tree
*** ## Reliability and Flow Control ### HTTP Client Configure upstream HTTP client connection settings:
Parameter Default Description
`--pool-idle-timeout-secs` 50 Idle timeout in seconds for pooled upstream HTTP connections. Can also be set with `SMG_POOL_IDLE_TIMEOUT_SECS`.
`--connect-timeout-secs` 10 Timeout in seconds for new upstream HTTP connections. Can also be set with `SMG_CONNECT_TIMEOUT_SECS`.
`--pool-max-idle-per-host` 500 Maximum idle upstream HTTP connections to keep per host. Can also be set with `SMG_POOL_MAX_IDLE_PER_HOST`.
`--tcp-keepalive-secs` 30 TCP keepalive idle time in seconds for upstream HTTP connections. Can also be set with `SMG_TCP_KEEPALIVE_SECS`.
### Retries Configure exponential backoff retries: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --retry-max-retries 5 \ --retry-initial-backoff-ms 50 \ --retry-max-backoff-ms 30000 \ --retry-backoff-multiplier 1.5 \ --retry-jitter-factor 0.2 ```
Parameter Default Description
`--retry-max-retries` 5 Maximum retry attempts
`--retry-initial-backoff-ms` 50 Initial backoff duration (ms)
`--retry-max-backoff-ms` 5000 Maximum backoff duration (ms)
`--retry-backoff-multiplier` 2.0 Exponential backoff multiplier
`--retry-jitter-factor` 0.1 Random jitter factor (0.0-1.0)
`--disable-retries` false Disable retries entirely
**Retryable Status Codes:** 408, 429, 500, 502, 503, 504 ### Circuit Breaker Per-worker circuit breakers prevent cascading failures: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --cb-failure-threshold 5 \ --cb-success-threshold 2 \ --cb-timeout-duration-secs 30 \ --cb-window-duration-secs 60 ```
Parameter Default Description
`--cb-failure-threshold` 5 Consecutive failures to open circuit
`--cb-success-threshold` 2 Successes to close from half-open
`--cb-timeout-duration-secs` 30 Time before half-open attempt
`--cb-window-duration-secs` 60 Failure counting window
`--disable-circuit-breaker` false Disable circuit breaker
**Circuit Breaker States:** * **Closed**: Normal operation, requests allowed * **Open**: Failing, requests rejected immediately * **Half-Open**: Testing recovery, limited requests allowed ### Rate Limiting and Queuing ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 http://worker2:8001 \ --max-concurrent-requests 256 \ --rate-limit-tokens-per-second 512 \ --queue-size 128 \ --queue-timeout-secs 30 ``` Requests beyond the concurrency limit wait in a FIFO queue. Returns: * `429 Too Many Requests` when queue is full * `408 Request Timeout` when queue timeout expires ### Health Checks ```bash Command theme={null} --health-check-interval-secs 30 \ --health-check-timeout-secs 10 \ --health-success-threshold 2 \ --health-failure-threshold 3 \ --health-check-endpoint /health ``` *** ## Reasoning Parser Integration The gateway includes built-in reasoning parsers for models that use Chain-of-Thought (CoT) reasoning with explicit thinking blocks. ### Supported Parsers
Parser ID Model Family Think Tokens
`deepseek-r1` DeepSeek-R1 `<think>...</think>` (initial reasoning)
`qwen3` Qwen-3 `<think>...</think>`
`qwen3-thinking` Qwen-3 Thinking `<think>...</think>` (initial reasoning)
`kimi` Kimi K2 Unicode think tokens
`glm45` GLM-4.5/4.6/4.7 `<think>...</think>`
`step3` Step-3 `<think>...</think>`
`minimax` MiniMax `<think>...</think>`
### Usage ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path deepseek-ai/DeepSeek-R1 \ --reasoning-parser deepseek-r1 ``` The gRPC router automatically: 1. Detects reasoning blocks in streaming output 2. Separates reasoning content from normal text 3. Applies incremental streaming parsing with buffer management 4. Handles partial token detection for correct streaming behavior *** ## Tool Call Parsing The gateway supports parsing function/tool calls from LLM outputs in multiple formats. ### Supported Formats
Parser Format Description
`json` JSON Standard JSON tool calls
`python` Pythonic Python function call syntax
`xml` XML XML-formatted tool calls
### Usage ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls grpc://127.0.0.1:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --tool-call-parser json ``` *** ## Tokenizer Management ### Tokenizer Sources The gateway supports multiple tokenizer backends: * **HuggingFace**: Load from HuggingFace Hub by model ID * **Local**: Load from local `tokenizer.json` or directory * **Tiktoken**: Auto-detect OpenAI GPT models (gpt-4, davinci, etc.) ### Configuration ```bash Command theme={null} # HuggingFace model --model-path meta-llama/Llama-3.1-8B-Instruct # Local tokenizer --tokenizer-path /path/to/tokenizer.json # With chat template override --chat-template /path/to/template.jinja ``` ### Tokenizer Caching Two-level caching for optimal performance:
Cache Type Description
L0 Exact match Whole-string caching for repeated prompts
L1 Prefix match Prefix boundary matching for incremental prompts
```bash Command theme={null} --enable-l0-cache \ --l0-max-entries 10000 \ --enable-l1-cache \ --l1-max-memory 52428800 # 50MB ``` *** ## MCP Integration The gateway provides native Model Context Protocol (MCP) client integration for tool execution. ### Supported Transports
Transport Description
STDIO Local process execution
SSE Server-Sent Events (HTTP)
Streamable Bidirectional streaming
### Configuration ```bash Command theme={null} python -m sglang_router.launch_router \ --mcp-config-path /path/to/mcp-config.yaml \ --worker-urls http://worker1:8000 ``` ### MCP Configuration File ```yaml Config theme={null} servers: - name: "filesystem" command: "npx" args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"] protocol: "stdio" required: false - name: "github" url: "https://api.github.com/mcp" token: "ghp_xxxxx" protocol: "sse" required: false - name: "custom-tools" url: "https://tools.example.com/mcp" protocol: "streamable" required: true pool: max_connections: 100 idle_timeout: 300 proxy: http: "http://proxy.internal:8080" https: "https://proxy.internal:8443" no_proxy: "localhost,127.0.0.1,*.internal" inventory: enable_refresh: true tool_ttl: 300 refresh_interval: 300 ``` *** ## Service Discovery (Kubernetes) Enable automatic worker discovery via Kubernetes pod selectors: ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker role=inference \ --service-discovery-namespace production \ --service-discovery-port 8000 ``` ### PD Mode Discovery ```bash Command theme={null} --pd-disaggregation \ --prefill-selector app=sglang component=prefill \ --decode-selector app=sglang component=decode \ --service-discovery ``` Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annotation. RBAC must allow `get`, `list`, and `watch` on pods. *** ## History and Data Connectors
Backend Description Usage
`memory` In-memory storage (default) `--history-backend memory`
`none` No persistence `--history-backend none`
`oracle` Oracle Autonomous Database `--history-backend oracle`
`postgres` PostgreSQL Database `--history-backend postgres`
`redis` Redis `--history-backend redis`
### Oracle Configuration ```bash Command theme={null} # Connection descriptor export ATP_DSN="(description=(address=(protocol=tcps)(port=1522)(host=adb.region.oraclecloud.com))(connect_data=(service_name=service_name)))" # Or TNS alias (requires wallet) export ATP_TNS_ALIAS="sglroutertestatp_high" export ATP_WALLET_PATH="/path/to/wallet" # Credentials export ATP_USER="admin" export ATP_PASSWORD="secret" export ATP_POOL_MIN=4 export ATP_POOL_MAX=32 python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend oracle ``` ### PostgreSQL Configuration ```bash Command theme={null} export POSTGRES_DB_URL="postgres://user:password@host:5432/dbname" python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend postgres ``` ### Redis Configuration ```bash Command theme={null} export REDIS_URL="redis://localhost:6379" export REDIS_POOL_MAX=16 export REDIS_RETENTION_DAYS=30 python -m sglang_router.launch_router \ --backend openai \ --worker-urls https://api.openai.com \ --history-backend redis \ --redis-retention-days 30 ``` Use `--redis-retention-days -1` for persistent storage (default is 30 days). *** ## WASM Middleware The gateway supports WebAssembly (WASM) middleware modules for custom request/response processing. This enables organization-specific logic for authentication, rate limiting, billing, logging, and more—without modifying or recompiling the gateway. ### Overview WASM middleware runs in a sandboxed environment with memory isolation, no network/filesystem access, and configurable resource limits.
Attach Point When Executed Use Cases
`OnRequest` Before forwarding to workers Auth, rate limiting, request modification
`OnResponse` After receiving worker response Logging, response modification, error handling
Action Description
`Continue` Proceed without modification
`Reject(status)` Reject request with HTTP status code
`Modify(...)` Modify headers, body, or status
### Examples Complete working examples are available in `examples/wasm/`:
Example Description
`auth/` API key authentication for protected routes
`rate_limit/` Per-client rate limiting (requests/minute)
`logging/` Request tracking headers and response modification
The interface definition is located at `src/wasm/interface`. ### Building Modules ```bash Command theme={null} # Prerequisites rustup target add wasm32-wasip2 cargo install wasm-tools # Build cargo build --target wasm32-wasip2 --release # Convert to component format wasm-tools component new \ target/wasm32-wasip2/release/my_middleware.wasm \ -o my_middleware.component.wasm ``` ### Deploying Modules ```bash Command theme={null} # Enable WASM support python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --enable-wasm # Upload module curl -X POST http://localhost:30000/wasm \ -H "Content-Type: application/json" \ -d '{ "modules": [{ "name": "auth-middleware", "file_path": "/absolute/path/to/auth.component.wasm", "module_type": "Middleware", "attach_points": [{"Middleware": "OnRequest"}] }] }' # List modules curl http://localhost:30000/wasm # Remove module curl -X DELETE http://localhost:30000/wasm/{module_uuid} ``` ### Runtime Configuration
Parameter Default Description
`max_memory_pages` 1024 (64MB) Maximum WASM memory
`max_execution_time_ms` 1000 Execution timeout
`max_stack_size` 1MB Stack size limit
`module_cache_size` 10 Cached modules per worker
**Note:** Rate limiting state is per-worker thread and not shared across gateway replicas. For production, consider implementing rate limiting at a shared layer (e.g., Redis) *** ## Language Bindings SGLang Model Gateway provides official language bindings for Python and Go, enabling integration with different technology stacks and organizational requirements. ### Python Bindings The Python bindings provide a PyO3-based wrapper around the Rust gateway library. This is a straightforward binding that calls the gateway server startup from Python. #### Installation ```bash Command theme={null} # From PyPI pip install sglang-router # Development build cd sgl-model-gateway/bindings/python pip install maturin && maturin develop --features vendored-openssl ``` #### Usage The Python bindings are used throughout this documentation. See the [Quick Start](#quick-start) and [Deployment Modes](#deployment-modes) sections for detailed examples. Key components: * `RouterArgs` dataclass with 50+ configuration options * `Router.from_args()` for programmatic startup * CLI commands: `smg launch`, `smg server`, `python -m sglang_router.launch_router` ### Go Bindings The Go bindings provide a high-performance gRPC client library for organizations with Go-based infrastructure. This is ideal for: * Integration with internal Go services and tooling * High-performance client applications * Building custom OpenAI-compatible proxy servers #### Architecture ```text Output theme={null} +-------------------------------------------+ | High-Level Go API | | (client.go - OpenAI-style interface) | +-------------------------------------------+ | gRPC Layer | +-------------------------------------------+ | Rust FFI Layer | | (Tokenization, Parsing, Conversion) | +-------------------------------------------+ ``` **Key Features:** * Native Rust tokenization via FFI (thread-safe, lock-free) * Full streaming support with context cancellation * Configurable channel buffer sizes for high concurrency * Built-in tool call parsing and chat template application #### Installation ```bash Command theme={null} # Build the FFI library first cd sgl-model-gateway/bindings/golang make build && make lib # Then use in your Go project go get github.com/sgl-project/sgl-go-sdk ``` **Requirements:** Go 1.24+, Rust toolchain #### Examples Complete working examples are available in `bindings/golang/examples/`:
Example Description
`simple/` Non-streaming chat completion
`streaming/` Streaming chat completion with SSE
`oai_server/` Full OpenAI-compatible HTTP server
```bash Command theme={null} # Run examples cd sgl-model-gateway/bindings/golang/examples/simple && ./run.sh cd sgl-model-gateway/bindings/golang/examples/streaming && ./run.sh cd sgl-model-gateway/bindings/golang/examples/oai_server && ./run.sh ``` #### Testing ```bash Command theme={null} cd sgl-model-gateway/bindings/golang # Unit tests go test -v ./... # Integration tests (requires running SGLang server) export SGL_GRPC_ENDPOINT=grpc://localhost:20000 export SGL_TOKENIZER_PATH=/path/to/tokenizer go test -tags=integration -v ./... ``` ### Comparison
Feature Python Go
**Primary Use** Gateway server launcher gRPC client library
**CLI Support** Full CLI (smg, sglang-router) Library only
**K8s Discovery** Native support N/A (client library)
**PD Mode** Built-in N/A (client library)
**When to Use Python:** Launching and managing the gateway server, service discovery, PD disaggregation. **When to Use Go:** Building custom client applications, integration with Go microservices, OpenAI-compatible proxy servers *** ## Security and Authentication ### Router API Key ```bash Command theme={null} python -m sglang_router.launch_router \ --api-key "your-router-api-key" \ --worker-urls http://worker1:8000 ``` Clients must supply `Authorization: Bearer ` for protected endpoints. ### Worker API Keys ```bash Command theme={null} # Add worker with explicit key curl -H "Authorization: Bearer router-key" \ -X POST http://localhost:8080/workers \ -H "Content-Type: application/json" \ -d '{"url":"http://worker:8000","api_key":"worker-key"}' ``` ### Security Configurations 1. **No Authentication** (default): Use only in trusted environments 2. **Router-only Authentication**: Clients authenticate to router 3. **Worker-only Authentication**: Router open, workers require keys 4. **Full Authentication**: Both router and workers protected ### TLS (HTTPS) for Gateway Server Enable TLS to serve the gateway over HTTPS: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --tls-cert-path /path/to/server.crt \ --tls-key-path /path/to/server.key ```
Parameter Description
`--tls-cert-path` Path to server certificate (PEM format)
`--tls-key-path` Path to server private key (PEM format)
Both parameters must be provided together. The gateway uses rustls with the ring crypto provider for TLS termination. If TLS is not configured, the gateway falls back to plain HTTP. ### mTLS for Worker Communication Enable mutual TLS (mTLS) for secure communication with workers in HTTP mode: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --client-cert-path /path/to/client.crt \ --client-key-path /path/to/client.key \ --ca-cert-path /path/to/ca.crt ```
Parameter Description
`--client-cert-path` Path to client certificate for mTLS (PEM format)
`--client-key-path` Path to client private key for mTLS (PEM format)
`--ca-cert-path` Path to CA certificate for verifying worker TLS (PEM format, repeatable)
**Key Points:** * Client certificate and key must be provided together * Multiple CA certificates can be added with multiple `--ca-cert-path` flags * Uses rustls backend when TLS is configured * Single HTTP client is created for all workers (assumes single security domain) * TCP keepalive (30 seconds) is enabled for long-lived connections ### Full TLS Configuration Example Gateway HTTPS + Worker mTLS + API Key authentication: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --tls-cert-path /etc/certs/server.crt \ --tls-key-path /etc/certs/server.key \ --client-cert-path /etc/certs/client.crt \ --client-key-path /etc/certs/client.key \ --ca-cert-path /etc/certs/ca.crt \ --api-key "secure-api-key" \ --policy cache_aware ``` *** ## Observability ### Prometheus Metrics Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`). #### Metric Categories (40+ metrics)
Layer Prefix Metrics
HTTP `smg_http_*` `requests_total`, `request_duration_seconds`, `responses_total`, `connections_active`, `rate_limit_total`
Router `smg_router_*` `requests_total`, `request_duration_seconds`, `request_errors_total`, `stage_duration_seconds`, `upstream_responses_total`
Inference `smg_router_*` `ttft_seconds`, `tpot_seconds`, `tokens_total`, `generation_duration_seconds`
Worker `smg_worker_*` `pool_size`, `connections_active`, `requests_active`, `health_checks_total`, `selection_total`, `errors_total`
Circuit Breaker `smg_worker_cb_*` `state`, `transitions_total`, `outcomes_total`, `consecutive_failures`, `consecutive_successes`
Retry `smg_worker_*` `retries_total`, `retries_exhausted_total`, `retry_backoff_seconds`
Discovery `smg_discovery_*` `registrations_total`, `deregistrations_total`, `sync_duration_seconds`, `workers_discovered`
MCP `smg_mcp_*` `tool_calls_total`, `tool_duration_seconds`, `servers_active`, `tool_iterations_total`
Database `smg_db_*` `operations_total`, `operation_duration_seconds`, `connections_active`, `items_stored`
#### Key Inference Metrics (gRPC mode)
Metric Type Description
`smg_router_ttft_seconds` Histogram Time to first token
`smg_router_tpot_seconds` Histogram Time per output token
`smg_router_tokens_total` Counter Total tokens (input/output)
`smg_router_generation_duration_seconds` Histogram End-to-end generation time
#### Duration Buckets 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 90s, 120s, 180s, 240s ### OpenTelemetry Tracing Enable distributed tracing with OTLP export: ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --enable-trace \ --otlp-traces-endpoint localhost:4317 ``` #### Features * OTLP/gRPC exporter (default port 4317) * W3C Trace Context propagation for HTTP and gRPC * Batch span processing (500ms delay, 64 span batch size) * Custom filtering to reduce noise * Trace context injection into upstream worker requests * Service name: `sgl-router` ### Logging ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --log-level debug \ --log-dir ./router_logs ``` Structured tracing with optional file sink. Log levels: `debug`, `info`, `warn`, `error`. ### Request ID Propagation ```bash Command theme={null} --request-id-headers x-request-id x-trace-id x-correlation-id ``` Responses include `x-request-id` header for correlation. *** ## Production Recommendations This section provides guidance for deploying SGLang Model Gateway in production environments. ### Security Best Practices **Always enable TLS in production:** ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls https://worker1:8443 https://worker2:8443 \ --tls-cert-path /etc/certs/server.crt \ --tls-key-path /etc/certs/server.key \ --client-cert-path /etc/certs/client.crt \ --client-key-path /etc/certs/client.key \ --ca-cert-path /etc/certs/ca.crt \ --api-key "${ROUTER_API_KEY}" ``` **Security Checklist:** * Enable TLS for gateway HTTPS termination * Enable mTLS for worker communication when workers are on untrusted networks * Set `--api-key` to protect router endpoints * Use Kubernetes Secrets or a secrets manager for credentials * Rotate certificates and API keys periodically * Restrict network access with firewalls or network policies ### High Availability **Scaling Strategy:** The gateway supports running multiple replicas behind a load balancer for high availability. However, there are important considerations:
Component Shared Across Replicas Impact
Worker Registry No (independent) Each replica discovers workers independently
Radix Cache Tree No (independent) Cache hits may decrease by 10-20%
Circuit Breaker State No (independent) Each replica tracks failures independently
Rate Limiting No (independent) Limits apply per-replica, not globally
**Recommendations:** 1. **Prefer horizontal scaling over vertical scaling**: Deploy multiple smaller gateway replicas rather than one large instance with excessive CPU and memory. This provides: * Better fault tolerance (single replica failure doesn't take down the gateway) * More predictable resource usage * Easier capacity planning 2. **Use Kubernetes Service Discovery**: Let the gateway automatically discover and manage workers: ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker \ --service-discovery-namespace production ``` 3. **Accept cache efficiency trade-off**: With multiple replicas, the cache-aware routing policy's radix tree is not synchronized across replicas. This means: * Each replica builds its own cache tree * Requests from the same user may hit different replicas * Expected cache hit rate reduction: **10-20%** * This is often acceptable given the HA benefits 4. **Configure session affinity (optional)**: If cache efficiency is critical, configure your load balancer for session affinity based on a consistent hash of the request (e.g., user ID or API key). **Example HA Architecture:** ```text Output theme={null} +-------------------+ | Load Balancer | | (L4/L7) | +---------+---------+ | +-------------------+-------------------+ | | | v v v +-----------+ +-----------+ +-----------+ | Gateway | | Gateway | | Gateway | | Replica 1 | | Replica 2 | | Replica 3 | +-----+-----+ +-----+-----+ +-----+-----+ | | | +-------------------+-------------------+ | +-------------------+-------------------+ | | | v v v +-----------+ +-----------+ +-----------+ | Worker | | Worker | | Worker | | Pod 1 | | Pod 2 | | Pod N | +-----------+ +-----------+ +-----------+ ``` ### Performance **Use gRPC mode for high throughput:** gRPC mode provides the highest performance for SGLang workers: ```bash Command theme={null} # Start workers in gRPC mode python -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --grpc-mode \ --port 20000 # Configure gateway for gRPC python -m sglang_router.launch_router \ --worker-urls grpc://worker1:20000 grpc://worker2:20000 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --policy cache_aware ``` **Performance Benefits of gRPC:** * Native Rust tokenization (no Python overhead) * Streaming with lower latency * Built-in reasoning parser execution * Tool call parsing in the gateway * Reduced serialization overhead **Tuning Recommendations:**
Parameter Recommendation Reason
`--policy` `cache_aware` Best for repeated prompts, \~30% latency reduction
`--max-concurrent-requests` 2-4x worker count Prevent overload while maximizing throughput
`--queue-size` 2x max-concurrent Buffer for burst traffic
`--request-timeout-secs` Based on max generation length Prevent stuck requests
### Kubernetes Deployment **Pod Labeling for Service Discovery:** For the gateway to discover workers automatically, label your worker pods consistently: ```yaml Config theme={null} # Worker Deployment (Regular Mode) apiVersion: apps/v1 kind: Deployment metadata: name: sglang-worker namespace: production spec: replicas: 4 selector: matchLabels: app: sglang-worker component: inference template: metadata: labels: app: sglang-worker component: inference model: llama-3-8b spec: containers: - name: worker image: lmsysorg/sglang:latest ports: - containerPort: 8000 name: http - containerPort: 20000 name: grpc ``` **Gateway configuration for discovery:** ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --selector app=sglang-worker component=inference \ --service-discovery-namespace production \ --service-discovery-port 8000 ``` **PD (Prefill/Decode) Mode Labeling:** ```yaml Config theme={null} # Prefill Worker metadata: labels: app: sglang-worker component: prefill annotations: sglang.ai/bootstrap-port: "9001" # Decode Worker metadata: labels: app: sglang-worker component: decode ``` **Gateway configuration for PD discovery:** ```bash Command theme={null} python -m sglang_router.launch_router \ --service-discovery \ --pd-disaggregation \ --prefill-selector app=sglang-worker component=prefill \ --decode-selector app=sglang-worker component=decode \ --service-discovery-namespace production ``` **RBAC Requirements:** The gateway needs permissions to watch pods: ```yaml Config theme={null} apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: sglang-gateway namespace: production rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"] *** apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sglang-gateway namespace: production subjects: - kind: ServiceAccount name: sglang-gateway namespace: production roleRef: kind: Role name: sglang-gateway apiGroup: rbac.authorization.k8s.io ``` ### Monitoring with PromQL Configure Prometheus to scrape the gateway metrics endpoint (default: `:29000/metrics`). **Essential Dashboards:** **1. Request Rate and Latency:** ```sql Example theme={null} # Request rate by endpoint sum(rate(smg_http_requests_total[5m])) by (path, method) # P50 latency histogram_quantile(0.50, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) # P99 latency histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) # Error rate sum(rate(smg_http_responses_total{status=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m])) ``` **2. Worker Health:** ```sql Example theme={null} # Healthy workers sum(smg_worker_pool_size) # Active connections per worker smg_worker_connections_active # Worker health check failures sum(rate(smg_worker_health_checks_total{result="failure"}[5m])) by (worker_id) ``` **3. Circuit Breaker Status:** ```sql Example theme={null} # Circuit breaker states (0=closed, 1=open, 2=half-open) smg_worker_cb_state # Circuit breaker transitions sum(rate(smg_worker_cb_transitions_total[5m])) by (worker_id, from_state, to_state) # Workers with open circuits count(smg_worker_cb_state == 1) ``` **4. Inference Performance (gRPC mode):** ```sql Example theme={null} # Time to first token (P50) histogram_quantile(0.50, sum(rate(smg_router_ttft_seconds_bucket[5m])) by (le, model)) # Time per output token (P99) histogram_quantile(0.99, sum(rate(smg_router_tpot_seconds_bucket[5m])) by (le, model)) # Token throughput sum(rate(smg_router_tokens_total[5m])) by (model, direction) # Generation duration P95 histogram_quantile(0.95, sum(rate(smg_router_generation_duration_seconds_bucket[5m])) by (le)) ``` **5. Rate Limiting and Queuing:** ```sql Example theme={null} # Rate limit rejections sum(rate(smg_http_rate_limit_total{decision="rejected"}[5m])) # Queue depth (if using concurrency limiting) smg_worker_requests_active # Retry attempts sum(rate(smg_worker_retries_total[5m])) by (worker_id) # Exhausted retries (failures after all retries) sum(rate(smg_worker_retries_exhausted_total[5m])) ``` **6. MCP Tool Execution:** ```sql Example theme={null} # Tool call rate sum(rate(smg_mcp_tool_calls_total[5m])) by (server, tool) # Tool latency P95 histogram_quantile(0.95, sum(rate(smg_mcp_tool_duration_seconds_bucket[5m])) by (le, tool)) # Active MCP server connections smg_mcp_servers_active ``` **Alerting Rules Example:** ```yaml Config theme={null} groups: - name: sglang-gateway rules: - alert: HighErrorRate expr: | sum(rate(smg_http_responses_total{status=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m])) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate on SGLang Gateway" - alert: CircuitBreakerOpen expr: count(smg_worker_cb_state == 1) > 0 for: 2m labels: severity: warning annotations: summary: "Worker circuit breaker is open" - alert: HighLatency expr: | histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) > 30 for: 5m labels: severity: warning annotations: summary: "P99 latency exceeds 30 seconds" - alert: NoHealthyWorkers expr: sum(smg_worker_pool_size) == 0 for: 1m labels: severity: critical annotations: summary: "No healthy workers available" ``` *** ## Configuration Reference ### Core Settings
Parameter Type Default Description
`--host` str 127.0.0.1 Router host
`--port` int 30000 Router port
`--worker-urls` list \[] Worker URLs (HTTP or gRPC)
`--policy` str cache\_aware Routing policy
`--max-concurrent-requests` int -1 Concurrency limit (-1 disables)
`--request-timeout-secs` int 600 Request timeout
`--max-payload-size` int 256MB Maximum request payload
### Prefill/Decode
Parameter Type Default Description
`--pd-disaggregation` flag false Enable PD mode
`--prefill` list \[] Prefill URLs + optional bootstrap ports
`--decode` list \[] Decode URLs
`--prefill-policy` str None Override policy for prefill nodes
`--decode-policy` str None Override policy for decode nodes
`--worker-startup-timeout-secs` int 600 Worker init timeout
### Kubernetes Discovery
Parameter Type Description
`--service-discovery` flag Enable discovery
`--selector` list Label selectors (key=value)
`--prefill-selector` / `--decode-selector` list PD mode selectors
`--service-discovery-namespace` str Namespace to watch
`--service-discovery-port` int Worker port (default 80)
`--bootstrap-port-annotation` str Annotation for bootstrap ports
### TLS Configuration
Parameter Type Description
`--tls-cert-path` str Server certificate for gateway HTTPS (PEM)
`--tls-key-path` str Server private key for gateway HTTPS (PEM)
`--client-cert-path` str Client certificate for worker mTLS (PEM)
`--client-key-path` str Client private key for worker mTLS (PEM)
`--ca-cert-path` str CA certificate for verifying workers (PEM, repeatable)
*** ## Troubleshooting ### Workers Never Ready Increase `--worker-startup-timeout-secs` or ensure health probes respond before router startup. ### Load Imbalance / Hot Workers Inspect `smg_router_requests_total` by worker and tune cache-aware thresholds (`--balance-*`, `--cache-threshold`). ### Circuit Breaker Flapping Increase `--cb-failure-threshold` or extend the timeout/window durations. Consider temporarily disabling retries. ### Queue Overflow (429) Increase `--queue-size` or reduce client concurrency. Ensure `--max-concurrent-requests` matches downstream capacity. ### Memory Growth Reduce `--max-tree-size` or lower `--eviction-interval-secs` for more aggressive cache pruning. ### Debugging ```bash Command theme={null} python -m sglang_router.launch_router \ --worker-urls http://worker1:8000 \ --log-level debug \ --log-dir ./router_logs ``` ### gRPC Connection Issues Ensure workers are started with `--grpc-mode` and verify `--model-path` or `--tokenizer-path` is provided to the router. ### Tokenizer Loading Failures Check HuggingFace Hub credentials (`HF_TOKEN` environment variable) for private models. Verify local paths are accessible. *** SGLang Model Gateway continues to evolve alongside the SGLang runtime. Keep CLI flags, integrations, and documentation aligned when adopting new features or contributing improvements. # SGLang for RL Systems Source: https://docs.sglang.io/docs/advanced_features/sglang_for_rl This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments. ## Why SGLang for RL Lifecycle? Let's embrace a guiding principle from early DeepMind's RL engineering: **Be a library, not a framework.** This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle: * **Fine-Grained Engine Sleep and Wake Up**: facilitate maximum-powered rollout and training * **Open-To-Use Refit Functionality**: diverse methods for co-location or disaggregation * **Easy To Postpone Generation**: enable partial rollout and dedicated rollout control * **Deterministic Inference**: achieve deterministic inference to enable zero training-inference mismatch * **Load Balancing Router**: cache-aware load-balancing for high-throughput rollout The following sections cover these aspects in detail. ## Fine-Grained Engine Sleep and Wake Up Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step. Under the hood, the RL team uses CUDA-graph-aware weight offload via [torch\_memory\_saver](https://github.com/fzyzcjy/torch_memory_saver) to preserve virtual memory addresses for graph replay. For details, see: [Efficient RL Training - Optimizing Memory Usage in verl](https://hebiao064.github.io/rl-memory-management). ### Server flag Enable memory saver support when launching the server: ```text Output theme={null} --enable-memory-saver ``` ### Release Memory **Endpoint:** `POST /release_memory_occupation` **Request body:**
Field Description Defaults Options
`tags` Which memory regions to release. If omitted, all are released. `None` Type: list\[str], values: `kv_cache`, `weights`
**Behavior notes:** * This call asserts there are no ongoing requests. Ensure the engine is idle before calling it. * If `kv_cache` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed. ### Resume Memory **Endpoint:** `POST /resume_memory_occupation` **Request body:**
Field Description Defaults Options
`tags` Which memory regions to resume. If omitted, all are resumed. `None` Type: list\[str], values: `kv_cache`, `weights`
## Open-To-Use Refit Functionality After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md). **How to choose:** * **From disk** is simplest and best for elastic rollout scaling and checkpointing. * **From tensor** is best for co-located training/rollout when you can pass in-memory tensors. * **From distributed** is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB). ### Update Weights from Disk **When to use:** * Save checkpoint to disk and update weights from disk * Dynamic scaling (new rollout instances can load from the same checkpoint) **Why it works well:** This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth. **Endpoint:** `POST /update_weights_from_disk` **Request body:**
Field Description Defaults Options
`model_path` The model path with the new weights. Required Type: str
`load_format` The format to load the weights. `None` Type: str
`abort_all_requests` Abort all running requests before update. `False` Type: bool
`weight_version` Optional weight version label tracked by the server. `None` Type: str
`is_async` Perform weight load asynchronously. `False` Type: bool
`torch_empty_cache` Empty torch cache. `False` Type: bool
`keep_pause` Keep scheduler paused after update. `False` Type: bool
`recapture_cuda_graph` Recapture CUDA graphs after update. `False` Type: bool
`token_step` Trainer step id for rollout bookkeeping. `0` Type: int
`flush_cache` Flush KV cache after update. `True` Type: bool
**Response body:**
Field Description Defaults Options
`success` Whether the update succeeded. - Type: bool
`message` Status / error message. - Type: str
`num_paused_requests` Number of paused requests during update. `0` Type: int
**Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)` **Diffusion engine (SGLang-Diffusion):** The diffusion engine exposes the same `POST /update_weights_from_disk` endpoint with the following behavior: * **All-or-nothing with rollback:** if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state. * **Offload-aware:** when layerwise offload (`--dit-layerwise-offload`) is enabled, the diffusion offload manager replaces GPU parameters with small `torch.empty((1,))` placeholders while real weights live in consolidated pinned CPU buffers. A naive `param.data.copy_()` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state. * **DTensor-aware:** parameters distributed via `torch.distributed.tensor` (tensor parallelism) are updated through `distribute_tensor` so that each shard is correctly placed on the right device mesh. **Request body:**
Field Description Defaults Options
model\_path The model path with the new weights. Required Type: str
flush\_cache Flush TeaCache state after update. True Type: bool
target\_modules List of module names to update (e.g. \["transformer"]). If omitted, all nn.Module components are updated. None Type: list\[str]
**Response body:**
Field Description Defaults Options
success Whether the update succeeded. - Type: bool
message Status / error message. - Type: str
> **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently. ### Update Weights from Tensor **When to use:** * Co-located training and rollout, where training can provide tensors directly * Fast in-memory updates **Important constraints:** This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts. **Endpoint:** `POST /update_weights_from_tensor` **Request body:**
Field Description Defaults Options
serialized\_named\_tensors Per-TP serialized tensor payloads. Required Type: list\[str|bytes]
load\_format Optional load format selector. None None, direct, flattened\_bucket, or a custom loader path string
flush\_cache Flush KV cache after update. True Type: bool
abort\_all\_requests Abort all running requests before update. False Type: bool
weight\_version Optional version label tracked by the server. None Type: str
**Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings. **Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)` ### Update Weights from Distributed Group **When to use:** * Disaggregated training and rollout * NCCL or IB-backed weight broadcast from training workers to rollout workers **How it works:** Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group. **Initialize weight update group** **Endpoint:** `POST /init_weights_update_group` **Request body:**
Field Description Defaults Options
`master_address` Group master address. Required Type: str
`master_port` Group master port. Required Type: int
`rank_offset` Offset for local rank mapping. Required Type: int
`world_size` Total world size. Required Type: int
`group_name` Group name. `weight_update_group` Type: str
`backend` Communication backend. `nccl` Type: str
**Update weight** **Endpoint:** `POST /update_weights_from_distributed` **Request body:**
Field Description Defaults Options
`names` Parameter names to update. Required Type: list\[str]
`dtypes` Dtype strings for each parameter. Required Type: list\[str]
`shapes` Tensor shapes. Required Type: list\[list\[int]]
`group_name` Group name. `weight_update_group` Type: str
`flush_cache` Flush KV cache after update. `True` Type: bool
`abort_all_requests` Abort all running requests before update. `False` Type: bool
`weight_version` Optional version label. `None` Type: str
`load_format` Optional format selector. `None` `None` or `flattened_bucket`
**Destroy weights update group** **Endpoint:** `POST /destroy_weights_update_group` **Request body:**
Field Description Defaults Options
`group_name` Group name. `weight_update_group` Type: str
**Python Engine APIs:** * `engine.init_weights_update_group(...)` * `engine.update_weights_from_distributed(names, dtypes, shapes, ...)` * `engine.destroy_weights_update_group(group_name)` ### Per-Token Weight Version Attribution A request can outlive a version change: the RL flow retracts it with `pause_generation`, refits, and continues it, or `POST /update_weight_version` relabels while it is still generating. `meta_info` then reports which tokens came from which weights: ```json Output theme={null} "weight_version": "42", "weight_versions": [ {"version": "41", "start": 0, "end": 57}, {"version": "42", "start": 57, "end": 128} ] ``` * Ranges are half-open `[start, end)` over output-token indices, prompt excluded. The usual case is a single span. ## Easy To Postpone Generation Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult. SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like [APRIL](https://arxiv.org/abs/2509.18521), terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work. `pause_generation` --- update weights --- `continue_generation` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks. ### Pause Generation **Endpoint:** `POST /pause_generation` **Request body:**
Field Description Defaults Options
`mode` Pause mode. `abort` `abort`, `retract`, `in_place`
**Modes:** * `abort`: Default behavior, identical to `abort` endpoint with `abort_all` set. Pending requests from `waiting_queue` and `running_queue` will be returned immediately to the caller. * `retract`: Put engine in "paused" state. Move running requests back to waiting queue. KV cache can be flushed and recomputed later. * `in_place`: Put engine in "paused" state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `flush_cache` call will be unsuccessful. ### Continue Generation **Endpoint:** `POST /continue_generation` ## Deterministic Inference In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem. SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md). **Server flag:** ```text Output theme={null} --enable-deterministic-inference ``` For more details, see [Deterministic Inference](./deterministic_inference) ## Load Balancing Router SGLang Model Gateway is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic. It has been deployed in the training of GLM 4.5+ models and proven to be highly efficient in production-level large-scale RL workloads. Key benefits for RL infrastructure: * **Async non-blocking efficiency**: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers. * **Elasticity and fault tolerance**: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption. * **Training–Inference alignment**: Using the SGLang Model Gateway for both training and inference ensures "What You See Is What You Get." This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment. * **Dynamic load balancing and long-tail mitigation**: Unlike static partitioning, the SGLang Model Gateway enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths. For deployment and configuration, see: [SGLang Model Gateway](./sgl_model_gateway) # Speculative Decoding Source: https://docs.sglang.io/docs/advanced_features/speculative_decoding SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines. ## Summary ### Jump to sections * [EAGLE Decoding](#eagle-decoding) * [EAGLE-2 Decoding](#eagle-2-decoding) * [EAGLE-2 Decoding with torch.compile](#eagle-2-decoding-with-torch-compile) * [EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling](#eagle-2-decoding-via-frequency-ranked-speculative-sampling) * [EAGLE-3 Decoding](#eagle-3-decoding) * [Multi Token Prediction](#multi-token-prediction) * [DFlash Decoding](#dflash-decoding) * [Standalone Speculative Decoding (Small Draft Model)](#standalone-speculative-decoding-small-draft-model) * [Speculative Decoding V2 (Overlap Scheduler)](#speculative-decoding-v2-overlap-scheduler) * [Ngram Speculative Decoding](#ngram-speculative-decoding) * [Full Parameter Reference](#full-parameter-reference) * [OOM Troubleshooting](#oom-troubleshooting) * [References](#references) ### Quick guidance * **Best speed/quality (recommended)**: Use **EAGLE-3** with `--speculative-algorithm EAGLE3`. * **Strong default / broad compatibility**: Use **EAGLE-2** with `--speculative-algorithm EAGLE`. * **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](./adaptive_speculative_decoding) on top of **EAGLE** with `--speculative-eagle-topk 1`. * **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`. * **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section). * **You have a DFlash draft checkpoint**: Use **DFLASH** with `--speculative-algorithm DFLASH` and `--speculative-draft-model-path ...`. * **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`). * **No extra model available**: Use **NGRAM** (`--speculative-algorithm NGRAM`, CUDA-only). ### Method comparison (mini table)
Method Draft source Separate draft model? How to enable Notes / constraints
EAGLE-2 EAGLE draft model (feature drafting + tree) Typically yes --speculative-algorithm EAGLE + --speculative-draft-model-path ... Tune --speculative-num-steps, --speculative-eagle-topk, --speculative-num-draft-tokens
EAGLE-2 + torch.compile Same as EAGLE-2 Typically yes Add --enable-torch-compile (optionally --torch-compile-max-bs) Benefit varies by hardware/model; benchmark to verify
EAGLE-2 + FR-Spec Same as EAGLE-2 + token subset Typically yes Add --speculative-token-map ... Reduces lm\_head overhead with high-frequency token vocab
EAGLE-3 EAGLE3 draft model Yes --speculative-algorithm EAGLE3 + --speculative-draft-model-path ... Best throughput in the benchmark below
MTP Built-in multi-token heads (model-specific) Often no See Multi Token Prediction section Uses speculative workflow; draft path may be auto-handled for some models
DFLASH DFlash draft model (linear block verification) Yes --speculative-algorithm DFLASH + --speculative-draft-model-path ... No --enable-dp-attention; pp\_size == 1; disables overlap scheduler & mixed chunked prefill
STANDALONE Smaller draft LLM (token-level) Yes --speculative-algorithm STANDALONE + --speculative-draft-model-path ... Does not support --enable-dp-attention
NGRAM Ngram cache from previous tokens No --speculative-algorithm NGRAM CUDA-only; no --enable-dp-attention; disables overlap scheduler & mixed chunked prefill
### Performance Highlights Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding. For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).
Method Throughput (tokens/s)
SGLang (w/o speculative, 1x H100) 158.34 tokens/s
SGLang + EAGLE-2 (1x H100) 244.10 tokens/s
SGLang + EAGLE-3 (1x H100) 373.25 tokens/s
*** ## EAGLE Decoding To enable EAGLE speculative decoding the following parameters are relevant:
Parameter Description Default
--speculative-draft-model-path Draft model path/weights. Typically required for EAGLE/EAGLE3 and STANDALONE. For some MTP-enabled models, this can be omitted. None
--speculative-num-steps Depth of autoregressive drafting. Increases speculation range but risks rejection cascades. Auto (5 for Llama/Grok; 3 for many other models)
--speculative-eagle-topk Branching factor per step. Improves candidate diversity and acceptance rate, but increases memory/compute consumption. Auto (4 for Llama/Grok; 1 for many other models)
--speculative-num-draft-tokens Maximum parallel verification capacity. Allows deeper tree evaluation but increases GPU memory usage. Auto (8 for Llama/Grok; 4 for many other models). If topk=1, it is adjusted to num\_steps + 1.
--speculative-accept-threshold-single Acceptance threshold for single-token verification. Lower values accept more aggressively. 1.0
--speculative-accept-threshold-acc Accumulated acceptance threshold across steps. 1.0
--speculative-attention-mode Attention mode for speculative operations (prefill or decode), affecting both target verification and draft extension. "prefill"
--speculative-draft-attention-backend Override attention backend for the draft model. None (same as target)
--speculative-draft-model-quantization Quantization method for the draft model. Use "unquant" to force no quantization even when the target model is quantized. Same as target model
--speculative-draft-model-revision Specific revision/commit of the draft model to load. None (auto-set to "main" when --speculative-draft-model-path is set and revision is omitted)
--speculative-draft-load-format Load format for the draft model weights. None
These parameters are mostly the same for EAGLE-2 and EAGLE-3. `--speculative-token-map` is ignored for EAGLE-3 models. For `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`: leave all three unset to use auto-tuning, or set all three explicitly when tuning. If you use EAGLE with `--speculative-eagle-topk 1` and your acceptance rate varies across requests, see [Adaptive Speculative Decoding](./adaptive_speculative_decoding). You can find the best combinations of these parameters with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). ### EAGLE-2 Decoding You can enable EAGLE-2 Decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model. **Launch the server:** ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-2-7b-chat-hf", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-2 Decoding with `torch.compile` You can optionally enable `torch.compile` to apply kernel-level optimizations (operator fusion, autotune) to the draft model. The actual speedup depends on your hardware, model architecture, and batch size. In some configurations (e.g., small draft models on H100 where cuBLAS is already optimal and CUDA graphs are enabled), the benefit may be negligible. We recommend benchmarking with and without this flag on your specific setup to verify whether it helps. To enable it, add `--enable-torch-compile` and optionally set `--torch-compile-max-bs`: ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-2-7b-chat-hf \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --enable-torch-compile \ --torch-compile-max-bs 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-2-7b-chat-hf", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling By employing a truncated high-frequency token vocabulary in the draft model, EAGLE speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, check out [the paper](https://arxiv.org/pdf/2502.14856). In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency tokens in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency tokens by directly downloading these tokens from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset). Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx). ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3-8B-Instruct \ --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ### EAGLE-3 Decoding You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model. ```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Multi Token Prediction We support [MTP (Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use `XiaomiMiMo/MiMo-7B-RL` as an example here (for DeepSeek MTP usage, refer to [DeepSeek-V3.2 cookbook §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2#4-2-3-multi-token-prediction-eagle-speculative-decoding)). ```bash Command theme={null} python3 -m sglang.launch_server \ --model XiaomiMiMo/MiMo-7B-RL \ --host 0.0.0.0 \ --trust-remote-code \ --speculative-algorithm EAGLE \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "XiaomiMiMo/MiMo-7B-RL", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` *** ## DFlash Decoding SGLang also supports **DFLASH** speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as `meta-llama/Llama-3.1-8B-Instruct` with `z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat`. Relevant parameters:
Parameter Description Default
--speculative-draft-model-path Required DFlash draft model path/weights. None
--speculative-num-draft-tokens DFlash verify block size. Inferred from draft config, otherwise 16
--speculative-dflash-block-size Alias of --speculative-num-draft-tokens for DFlash. None
--speculative-dflash-draft-window-size Draft KV sliding-window size. Must be >= speculative-num-draft-tokens when set. None
```bash Command theme={null} python3 -m sglang.launch_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --speculative-algorithm DFLASH \ --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "Write a quicksort implementation in Python."}, ], temperature=0, max_tokens=128, ) print(response.choices[0].message.content) ``` *** ## Standalone Speculative Decoding (Small Draft Model) Besides EAGLE/MTP, SGLang also supports **token-level speculative decoding** using a smaller **draft model**. Enable it with `--speculative-algorithm STANDALONE` and provide a draft model via `--speculative-draft-model-path`. Relevant parameters:
Parameter Description Default
--speculative-draft-model-path Draft model weights (smaller than the target model). None
--speculative-num-steps Draft depth (how many steps the draft model runs autoregressively). 3 (auto default for STANDALONE)
--speculative-eagle-topk Branching factor (token candidates per step). 1 (auto default for STANDALONE)
--speculative-num-draft-tokens Verification capacity. 4 (auto default for STANDALONE)
--speculative-draft-model-quantization Quantization for the draft model. Use "unquant" to disable quantization on the draft even when the target is quantized. Same as target
> **Note:** Standalone speculative decoding currently **does not support** `--enable-dp-attention`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm STANDALONE \ --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \ --speculative-num-steps 4 \ --speculative-eagle-topk 2 \ --speculative-num-draft-tokens 7 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Speculative Decoding V2 (Overlap Scheduler) Speculative decoding runs the V2 speculative workers (e.g. `StandaloneWorkerV2`, `EAGLEWorkerV2`) with the overlap scheduler enabled by default. Pass `--disable-overlap-schedule` to fall back to the synchronous (non-overlap) path. Notes: * The overlap scheduler currently only supports `--speculative-eagle-topk 1`; **set `--speculative-eagle-topk 1` explicitly**. * If you explicitly set `--speculative-eagle-topk > 1`, the server will error. * If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama). This is incompatible with the overlap scheduler and may not always trigger an immediate config error, so set `--speculative-eagle-topk 1` explicitly. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm STANDALONE \ --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Ngram Speculative Decoding SGLang also supports **ngram-based speculative decoding** (no separate draft model). It retrieves draft tokens from an ngram cache built from previously generated tokens, and then verifies them with the target model. Enable it with: * `--speculative-algorithm NGRAM` ### Ngram-specific parameters
Parameter Description Default
--speculative-num-draft-tokens Number of draft tokens verified per step. If omitted, defaults to min(--speculative-ngram-max-trie-depth, 12). 12 (with default ngram settings)
--speculative-ngram-min-bfs-breadth Minimum BFS breadth. 1
--speculative-ngram-max-bfs-breadth Maximum BFS breadth. 10
--speculative-ngram-match-type Ngram tree-building mode: "BFS" for recency-based expansion or "PROB" for frequency-based expansion. "BFS"
--speculative-ngram-max-trie-depth Maximum suffix length stored and matched by the ngram trie. 18
--speculative-ngram-capacity Cache capacity (number of entries). 10,000,000
Notes: * Ngram speculative decoding **only supports CUDA**. * It currently **does not support** `--enable-dp-attention`. * It disables the overlap scheduler and mixed chunked prefill. * If `--speculative-ngram-max-bfs-breadth > 1` (thus `speculative_eagle_topk > 1`) and `page_size > 1`, use `--attention-backend flashinfer`; otherwise the server will error. * Optional: set `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` to force greedy verification. ```bash Command theme={null} python3 -m sglang.launch_server \ --model Qwen/Qwen2.5-7B-Instruct \ --speculative-algorithm NGRAM \ --speculative-num-draft-tokens 16 \ --speculative-ngram-max-bfs-breadth 10 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --log-level warning ``` **Send a request:** ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-7B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` *** ## Full Parameter Reference Below is a comprehensive list of all speculative decoding parameters available in SGLang: ### Core parameters
Parameter Type Default Description
--speculative-algorithm str None Algorithm to use: DFLASH, EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE)
--speculative-draft-model-path str None Path to the draft model weights
--speculative-draft-model-revision str None Specific revision/commit of the draft model ("main" is auto-used when draft path is set and revision is omitted)
--speculative-draft-load-format str None Load format for draft model weights
--speculative-num-steps int None (auto-chosen when omitted) Autoregressive drafting depth
--speculative-eagle-topk int None (auto-chosen when omitted) Branching factor per drafting step
--speculative-num-draft-tokens int None (auto-chosen when omitted) Maximum number of draft tokens for verification
--speculative-dflash-block-size int None DFlash-only alias of --speculative-num-draft-tokens
--speculative-dflash-draft-window-size int None DFlash-only draft KV sliding-window size
--speculative-accept-threshold-single float 1.0 Single-token acceptance threshold
--speculative-accept-threshold-acc float 1.0 Accumulated acceptance threshold
--speculative-token-map str None Path to FR-Spec high-frequency token map
--speculative-attention-mode str "prefill" Attention mode for speculative operations ("prefill" or "decode")
--speculative-draft-attention-backend str None Override attention backend for the draft model
--speculative-moe-runner-backend str None MoE runner backend for the draft model
--speculative-moe-a2a-backend str None MoE all-to-all backend for the draft model
--speculative-draft-model-quantization str Same as target Quantization for the draft model ("unquant" to disable)
### Ngram-specific parameters
Parameter Type Default Description
--speculative-ngram-min-bfs-breadth int 1 Minimum BFS breadth
--speculative-ngram-max-bfs-breadth int 10 Maximum BFS breadth
--speculative-ngram-match-type str "BFS" Ngram tree-building mode: "BFS" for recency-based expansion or "PROB" for frequency-based expansion
--speculative-ngram-max-trie-depth int 18 Maximum suffix length stored and matched by the ngram trie
--speculative-ngram-capacity int 10,000,000 Cache capacity
### Environment variables
Variable Default Description
SGLANG\_NGRAM\_FORCE\_GREEDY\_VERIFY False Force greedy verification for ngram decoding
### Other related flags
Parameter Description
--enable-multi-layer-eagle Enable multi-layer EAGLE (auto-enabled for MiMoV2 and Step3p5 models)
--enable-torch-compile Enable torch.compile for kernel-level optimizations
--torch-compile-max-bs Maximum batch size for torch.compile
*** ## OOM Troubleshooting > \[!WARNING] > **Out of Memory (OOM)?** Speculative decoding may increase GPU memory usage because the draft tree, CUDA graphs, and verification-related buffers consume additional VRAM. If you encounter OOM errors, try the following adjustments. ### Step 1: Lower static memory fraction (most effective) ```bash Command theme={null} --mem-fraction-static 0.5 # when omitted, this value is auto-computed ``` * `--mem-fraction-static` controls the memory budget for model weights + KV cache pool. * Lowering it directly increases dynamic headroom for activations and CUDA graph buffers. * If omitted, SGLang auto-estimates this value from other settings, and those auto settings can still be too aggressive for some workloads. ### Step 2: Reduce CUDA graph batch size ```bash Command theme={null} # Fewer CUDA graph captures = less memory reserved --cuda-graph-max-bs-decode 4 # or even 2 for tight memory situations ``` * If omitted, `--cuda-graph-max-bs-decode` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs. ### Step 3: Reduce draft tree size These three parameters directly control how much memory the draft tree consumes: ```bash Command theme={null} # Before (aggressive, high memory) --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 # After (conservative, lower memory) --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ``` ### Step 4: Limit concurrent requests ```bash Command theme={null} # Fewer concurrent requests lowers in-flight load and can reduce OOM risk --max-running-requests 4 ``` ### Quick OOM recovery recipe If you're hitting OOM and just want something that works, start with this minimal configuration and scale up: ```bash Command theme={null} python3 -m sglang.launch_server \ --model \ --speculative-algorithm EAGLE \ --speculative-draft-model-path \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --cuda-graph-max-bs-decode 2 \ --mem-fraction-static 0.5 \ --max-running-requests 4 \ --log-level warning ``` Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs-decode`. Increase `--mem-fraction-static` last, only after the run is stable. *** ## References EAGLE process is as follows: * Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$. * The next token is then sampled from $p_{k+2}=\text{LMHead}(f_{k+1})$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again. * In SGLang's EAGLE-2 implementation, the draft tree is expanded for the configured steps and then reranked to select the top `speculative_num_draft_tokens` final nodes as draft tokens. * EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner. This enhances drafting accuracy by operating on features instead of tokens for more regular inputs and by additionally passing tokens from the next timestep to reduce sampling randomness. For more details, see the [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) papers. For guidance on how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train). For EAGLE-3 training specifically, check out [SpecForge](https://github.com/sgl-project/SpecForge), the SGLang team's training framework designed for EAGLE-3 speculative decoding models with seamless porting to SGLang serving. See the [SpecForge documentation](https://docs.sglang.ai/SpecForge/) and [blog post](https://lmsys.org/blog/2025-07-25-spec-forge) for details. # Structured Outputs Source: https://docs.sglang.io/docs/advanced_features/structured_outputs You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request. SGLang supports three grammar backends: * [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints. * [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints. * [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints. We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar). To use Outlines, simply add `--grammar-backend outlines` when launching the server. To use llguidance, add `--grammar-backend llguidance` when launching the server. If no backend is specified, XGrammar will be used as the default. For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'. ## OpenAI Compatible API ```python Example theme={null} import openai import os from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process os.environ["TOKENIZERS_PARALLELISM"] = "false" server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}") client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") ``` ### JSON you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response. **Using Pydantic** ```python Example theme={null} from pydantic import BaseModel, Field # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ { "role": "user", "content": "Please generate the information of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=128, response_format={ "type": "json_schema", "json_schema": { "name": "foo", # convert the pydantic model to json schema "schema": CapitalInfo.model_json_schema(), }, }, ) response_content = response.choices[0].message.content # validate the JSON response by the pydantic model capital_info = CapitalInfo.model_validate_json(response_content) print_highlight(f"Validated response: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} import json json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ { "role": "user", "content": "Give me the information of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=128, response_format={ "type": "json_schema", "json_schema": {"name": "foo", "schema": json.loads(json_schema)}, }, ) print_highlight(response.choices[0].message.content) ``` ### EBNF ```python Example theme={null} ebnf_grammar = """ root ::= city | description city ::= "London" | "Paris" | "Berlin" | "Rome" description ::= city " is " status status ::= "the capital of " country country ::= "England" | "France" | "Germany" | "Italy" """ response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "system", "content": "You are a helpful geography bot."}, { "role": "user", "content": "Give me the information of the capital of France.", }, ], temperature=0, max_tokens=32, extra_body={"ebnf": ebnf_grammar}, ) print_highlight(response.choices[0].message.content) ``` ### Regular expression ```python Example theme={null} response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "What is the capital of France?"}, ], temperature=0, max_tokens=128, extra_body={"regex": "(Paris|London)"}, ) print_highlight(response.choices[0].message.content) ``` ### Structural Tag ```python Example theme={null} tool_get_current_weather = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } tool_get_current_date = { "type": "function", "function": { "name": "get_current_date", "description": "Get the current date and time for a given timezone", "parameters": { "type": "object", "properties": { "timezone": { "type": "string", "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'", } }, "required": ["timezone"], }, }, } schema_get_current_weather = tool_get_current_weather["function"]["parameters"] schema_get_current_date = tool_get_current_date["function"]["parameters"] def get_messages(): return [ { "role": "system", "content": f""" # Tool Instructions - Always execute python code in messages that you share. - When looking for real time information use relevant functions if available else fallback to brave_search You have access to the following functions: Use the function 'get_current_weather' to: Get the current weather in a given location {tool_get_current_weather["function"]} Use the function 'get_current_date' to: Get the current date and time for a given timezone {tool_get_current_date["function"]} If a you choose to call a function ONLY reply in the following format: <{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}} where start_tag => ` a JSON dict with the function argument name as key and function argument value as value. end_tag => `` Here is an example, {{"example_name": "example_value"}} Reminder: - Function calls MUST follow the specified format - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - Always add your sources when using search results to answer the user query You are a helpful assistant.""", }, { "role": "user", "content": "You are in New York. Please get the current date and time, and the weather.", }, ] messages = get_messages() response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=messages, response_format={ "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, }, ) print_highlight(response.choices[0].message.content) ``` ## Native API and SGLang Runtime (SRT) ### JSON **Using Pydantic** ```python Example theme={null} import requests import json from pydantic import BaseModel, Field from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct") # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") # Make API request messages = [ { "role": "user", "content": "Here is the information of the capital of France in the JSON format.\n", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json.dumps(CapitalInfo.model_json_schema()), }, }, ) print_highlight(response.json()) response_data = json.loads(response.json()["text"]) # validate the response by the pydantic model capital_info = CapitalInfo.model_validate(response_data) print_highlight(f"Validated response: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) # JSON response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json_schema, }, }, ) print_highlight(response.json()) ``` ### EBNF ```python Example theme={null} messages = [ { "role": "user", "content": "Give me the information of the capital of France.", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "max_new_tokens": 128, "temperature": 0, "n": 3, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), }, "stream": False, "return_logprob": False, }, ) print_highlight(response.json()) ``` ### Regular expression ```python Example theme={null} messages = [ { "role": "user", "content": "Paris is the capital of", } ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "sampling_params": { "temperature": 0, "max_new_tokens": 64, "regex": "(France|England)", }, }, ) print_highlight(response.json()) ``` ### Structural Tag ```python Example theme={null} from transformers import AutoTokenizer # generate an answer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct") text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) payload = { "text": text, "sampling_params": { "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, } ) }, } # Send POST request to the API endpoint response = requests.post(f"http://localhost:{port}/generate", json=payload) print_highlight(response.json()) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Offline Engine API ```python Example theme={null} import sglang as sgl llm = sgl.Engine( model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", grammar_backend="xgrammar" ) ``` ### JSON **Using Pydantic** ```python Example theme={null} import json from pydantic import BaseModel, Field prompts = [ "Give me the information of the capital of China in the JSON format.", "Give me the information of the capital of France in the JSON format.", "Give me the information of the capital of Ireland in the JSON format.", ] # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") sampling_params = { "temperature": 0.1, "top_p": 0.95, "json_schema": json.dumps(CapitalInfo.model_json_schema()), } outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}") # validate the output by the pydantic model capital_info = CapitalInfo.model_validate_json(output["text"]) print_highlight(f"Validated output: {capital_info.model_dump_json()}") ``` **JSON Schema Directly** ```python Example theme={null} prompts = [ "Give me the information of the capital of China in the JSON format.", "Give me the information of the capital of France in the JSON format.", "Give me the information of the capital of Ireland in the JSON format.", ] json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) sampling_params = {"temperature": 0.1, "top_p": 0.95, "json_schema": json_schema} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### EBNF ```python Example theme={null} prompts = [ "Give me the information of the capital of France.", "Give me the information of the capital of Germany.", "Give me the information of the capital of Italy.", ] sampling_params = { "temperature": 0.8, "top_p": 0.95, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), } outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Regular expression ```python Example theme={null} prompts = [ "Please provide information about London as a major global city:", "Please provide information about Paris as a major global city:", ] sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Structural Tag ```python Example theme={null} text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) prompts = [text] sampling_params = { "temperature": 0.8, "top_p": 0.95, "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "content": { "type": "json_schema", "json_schema": schema_get_current_weather, }, "end": "", }, { "begin": "", "content": { "type": "json_schema", "json_schema": schema_get_current_date, }, "end": "", }, ], "at_least_one": False, "stop_after_first": False, }, } ), } # Send POST request to the API endpoint outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print_highlight("===============================") print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ```python Example theme={null} llm.shutdown() ``` # Structured Outputs For Reasoning Models Source: https://docs.sglang.io/docs/advanced_features/structured_outputs_for_reasoning_models When working with reasoning models that use special tokens like `...` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output. SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output. To enable this feature, use the `--reasoning-parser` flag which decide the think\_end\_token, such as ``, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag. ## Supported Models Currently, SGLang supports the following reasoning models: * [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `` and `` tags. * [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `` and `` tags. ## Usage ## OpenAI Compatible API Specify the `--grammar-backend`, `--reasoning-parser` option. ```python Example theme={null} import openai import os from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process os.environ["TOKENIZERS_PARALLELISM"] = "false" server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning" ) wait_for_server(f"http://localhost:{port}") client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") ``` ### JSON you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response. **Using Pydantic** ```python Example theme={null} from pydantic import BaseModel, Field # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, response_format={ "type": "json_schema", "json_schema": { "name": "foo", # convert the pydantic model to json schema "schema": CapitalInfo.model_json_schema(), }, }, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` **JSON Schema Directly** ```python Example theme={null} import json json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, response_format={ "type": "json_schema", "json_schema": {"name": "foo", "schema": json.loads(json_schema)}, }, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### EBNF ```python Example theme={null} ebnf_grammar = """ root ::= city | description city ::= "London" | "Paris" | "Berlin" | "Rome" description ::= city " is " status status ::= "the capital of " country country ::= "England" | "France" | "Germany" | "Italy" """ response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ {"role": "system", "content": "You are a helpful geography bot."}, { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ], temperature=0, max_tokens=2048, extra_body={"ebnf": ebnf_grammar}, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### Regular expression ```python Example theme={null} response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=[ {"role": "assistant", "content": "What is the capital of France?"}, ], temperature=0, max_tokens=2048, extra_body={"regex": "(Paris|London)"}, ) print_highlight( f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}" ) ``` ### Structural Tag ```python Example theme={null} tool_get_current_weather = { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } tool_get_current_date = { "type": "function", "function": { "name": "get_current_date", "description": "Get the current date and time for a given timezone", "parameters": { "type": "object", "properties": { "timezone": { "type": "string", "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'", } }, "required": ["timezone"], }, }, } schema_get_current_weather = tool_get_current_weather["function"]["parameters"] schema_get_current_date = tool_get_current_date["function"]["parameters"] def get_messages(): return [ { "role": "system", "content": f""" # Tool Instructions - Always execute python code in messages that you share. - When looking for real time information use relevant functions if available else fallback to brave_search You have access to the following functions: Use the function 'get_current_weather' to: Get the current weather in a given location {tool_get_current_weather["function"]} Use the function 'get_current_date' to: Get the current date and time for a given timezone {tool_get_current_date["function"]} If a you choose to call a function ONLY reply in the following format: <{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}} where start_tag => ` a JSON dict with the function argument name as key and function argument value as value. end_tag => `` Here is an example, {{"example_name": "example_value"}} Reminder: - Function calls MUST follow the specified format - Required parameters MUST be specified - Only call one function at a time - Put the entire function call reply on one line - Always add your sources when using search results to answer the user query You are a helpful assistant.""", }, { "role": "assistant", "content": "You are in New York. Please get the current date and time, and the weather.", }, ] messages = get_messages() response = client.chat.completions.create( model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", messages=messages, response_format={ "type": "structural_tag", "max_new_tokens": 2048, "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": [" Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API. ### JSON **Using Pydantic** ```python Example theme={null} import requests from pydantic import BaseModel, Field from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B") # Define the schema using Pydantic class CapitalInfo(BaseModel): name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city") population: int = Field(..., description="Population of the capital city") messages = [ { "role": "assistant", "content": "Give me the information and population of the capital of France in the JSON format.", }, ] text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) # Make API request response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "json_schema": json.dumps(CapitalInfo.model_json_schema()), }, }, ) print(response.json()) reasoing_content = response.json()["text"].split("")[0] content = response.json()["text"].split("")[1] print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}") ``` **JSON Schema Directly** ```python Example theme={null} json_schema = json.dumps( { "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], } ) # JSON text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) response = requests.post( f"http://localhost:{port}/generate", json={ "text": text, "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "json_schema": json_schema, }, }, ) print_highlight(response.json()) ``` ### EBNF ```python Example theme={null} response = requests.post( f"http://localhost:{port}/generate", json={ "text": "Give me the information of the capital of France.", "require_reasoning": True, "sampling_params": { "max_new_tokens": 2048, "temperature": 0, "n": 3, "ebnf": ( "root ::= city | description\n" 'city ::= "London" | "Paris" | "Berlin" | "Rome"\n' 'description ::= city " is " status\n' 'status ::= "the capital of " country\n' 'country ::= "England" | "France" | "Germany" | "Italy"' ), }, "stream": False, "return_logprob": False, }, ) print(response.json()) ``` ### Regular expression ```python Example theme={null} response = requests.post( f"http://localhost:{port}/generate", json={ "text": "Paris is the capital of", "require_reasoning": True, "sampling_params": { "temperature": 0, "max_new_tokens": 2048, "regex": "(France|England)", }, }, ) print(response.json()) ``` ### Structural Tag ```python Example theme={null} text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, return_dict=False ) payload = { "text": text, "require_reasoning": True, "sampling_params": { "max_new_tokens": 2048, "structural_tag": json.dumps( { "type": "structural_tag", "structures": [ { "begin": "", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": ["", "schema": schema_get_current_weather, "end": "", }, { "begin": "", "schema": schema_get_current_date, "end": "", }, ], "triggers": [" Parser Supported Models Notes `apertus2509` Apertus 2509 (e.g., `swiss-ai/Apertus-{8,70}B-Instruct-2509`) Tool calls are emitted as a JSON list of single-key objects: `<|tools_prefix|>[{"tool": {...}}]<|tools_suffix|>`. `deepseekv3` DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`) Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command. `deepseekv31` DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`) Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command. `deepseekv32` DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`) `glm` GLM series (e.g. `zai-org/GLM-4.6`) `gpt-oss` GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`) The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role="tool"` messages, which enables the model to generate the final content. `kimi_k2` `moonshotai/Kimi-K2-Instruct` `llama3` Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`) `llama4` Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`) `mistral` Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`) `pythonic` Llama-3.2 / Llama-3.3 / Llama-4 Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template. `qwen` Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder `qwen3_coder` Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`) `step3` Step-3 ## OpenAI Compatible API ### Launching the Server ```python Example theme={null} import json from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process from openai import OpenAI server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning" # qwen25 ) wait_for_server(f"http://localhost:{port}") ``` Note that `--tool-call-parser` defines the parser used to interpret responses. ### Define Tools for Function Call Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters. ```python Example theme={null} # Define tools tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "state": { "type": "string", "description": "the two-letter abbreviation for the state that the city is" " in, e.g. 'CA' which would mean 'California'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "state", "unit"], }, }, } ] ``` ### Define Messages ```python Example theme={null} def get_messages(): return [ { "role": "user", "content": "What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.", } ] messages = get_messages() ``` ### Initialize the Client ```python Example theme={null} # Initialize OpenAI-like client client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1") model_name = client.models.list().data[0].id ``` ### Non-Streaming Request ```python Example theme={null} # Non-streaming mode test response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, max_tokens=1024, stream=False, # Non-streaming tools=tools, ) print_highlight("Non-stream response:") print_highlight(response_non_stream) print_highlight("==== content ====") print_highlight(response_non_stream.choices[0].message.content) print_highlight("==== tool_calls ====") print_highlight(response_non_stream.choices[0].message.tool_calls) ``` #### Handle Tools When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly. ```python Example theme={null} name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name arguments_non_stream = ( response_non_stream.choices[0].message.tool_calls[0].function.arguments ) print_highlight(f"Final streamed function call name: {name_non_stream}") print_highlight(f"Final streamed function call arguments: {arguments_non_stream}") ``` ### Streaming Request ```python Example theme={null} # Streaming mode test print_highlight("Streaming response:") response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, max_tokens=1024, stream=True, # Enable streaming tools=tools, ) texts = "" tool_calls = [] name = "" arguments = "" for chunk in response_stream: if chunk.choices[0].delta.content: texts += chunk.choices[0].delta.content if chunk.choices[0].delta.tool_calls: tool_calls.append(chunk.choices[0].delta.tool_calls[0]) print_highlight("==== Text ====") print_highlight(texts) print_highlight("==== Tool Call ====") for tool_call in tool_calls: print_highlight(tool_call) ``` #### Handle Tools When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly. ```python Example theme={null} # Parse and combine function call arguments arguments = [] for tool_call in tool_calls: if tool_call.function.name: print_highlight(f"Streamed function call name: {tool_call.function.name}") if tool_call.function.arguments: arguments.append(tool_call.function.arguments) # Combine all fragments into a single JSON string full_arguments = "".join(arguments) print_highlight(f"streamed function call arguments: {full_arguments}") ``` ### Define a Tool Function ```python Example theme={null} # This is a demonstration, define real function according to your usage. def get_current_weather(city: str, state: str, unit: "str"): return ( f"The weather in {city}, {state} is 85 degrees {unit}. It is " "partly cloudly, with highs in the 90's." ) available_tools = {"get_current_weather": get_current_weather} ``` ### Execute the Tool ```python Example theme={null} messages.append(response_non_stream.choices[0].message) # Call the corresponding tool function tool_call = messages[-1].tool_calls[0] tool_name = tool_call.function.name tool_to_call = available_tools[tool_name] result = tool_to_call(**(json.loads(tool_call.function.arguments))) print_highlight(f"Function call result: {result}") # messages.append({"role": "tool", "content": result, "name": tool_name}) messages.append( { "role": "tool", "tool_call_id": tool_call.id, "content": str(result), "name": tool_name, } ) print_highlight(f"Updated message history: {messages}") ``` ### Send Results Back to Model ```python Example theme={null} final_response = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.95, stream=False, tools=tools, ) print_highlight("Non-stream response:") print_highlight(final_response) print_highlight("==== Text ====") print_highlight(final_response.choices[0].message.content) ``` ## Native API and SGLang Runtime (SRT) ```python Example theme={null} from transformers import AutoTokenizer import requests # generate an answer tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct") messages = get_messages() input = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False ) gen_url = f"http://localhost:{port}/generate" gen_data = { "text": input, "sampling_params": { "skip_special_tokens": False, "max_new_tokens": 1024, "temperature": 0, "top_p": 0.95, }, } gen_response = requests.post(gen_url, json=gen_data).json()["text"] print_highlight("==== Response ====") print_highlight(gen_response) # parse the response parse_url = f"http://localhost:{port}/parse_function_call" function_call_input = { "text": gen_response, "tool_call_parser": "qwen25", "tools": tools, } function_call_response = requests.post(parse_url, json=function_call_input) function_call_response_json = function_call_response.json() print_highlight("==== Text ====") print(function_call_response_json["normal_text"]) print_highlight("==== Calls ====") print("function name: ", function_call_response_json["calls"][0]["name"]) print("function arguments: ", function_call_response_json["calls"][0]["parameters"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Offline Engine API ```python Example theme={null} import sglang as sgl from sglang.srt.function_call.function_call_parser import FunctionCallParser from sglang.srt.managers.io_struct import Tool, Function llm = sgl.Engine(model_path="Qwen/Qwen2.5-7B-Instruct") tokenizer = llm.tokenizer_manager.tokenizer input_ids = tokenizer.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False ) # Note that for gpt-oss tool parser, adding "no_stop_trim": True # to make sure the tool call token is not trimmed. sampling_params = { "max_new_tokens": 1024, "temperature": 0, "top_p": 0.95, "skip_special_tokens": False, } # 1) Offline generation result = llm.generate(input_ids=input_ids, sampling_params=sampling_params) generated_text = result["text"] # Assume there is only one prompt print_highlight("=== Offline Engine Output Text ===") print_highlight(generated_text) # 2) Parse using FunctionCallParser def convert_dict_to_tool(tool_dict: dict) -> Tool: function_dict = tool_dict.get("function", {}) return Tool( type=tool_dict.get("type", "function"), function=Function( name=function_dict.get("name"), description=function_dict.get("description"), parameters=function_dict.get("parameters"), ), ) tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools] parser = FunctionCallParser(tools=tools, tool_call_parser="qwen25") normal_text, calls = parser.parse_non_stream(generated_text) print_highlight("=== Parsing Result ===") print("Normal text portion:", normal_text) print_highlight("Function call portion:") for call in calls: # call: ToolCallItem print_highlight(f" - tool name: {call.name}") print_highlight(f" parameters: {call.parameters}") # 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc. ``` ```python Example theme={null} llm.shutdown() ``` ## Tool Choice Mode SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior. ### Supported Tool Choice Options * **`tool_choice="required"`**: Forces the model to call at least one tool * **`tool_choice={"type": "function", "function": {"name": "specific_function"}}`**: Forces the model to call a specific function ### Backend Compatibility Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`. ### Example: Required Tool Choice ```python Example theme={null} from openai import OpenAI from sglang.utils import wait_for_server, print_highlight, terminate_process from sglang.test.doc_patch import launch_server_cmd # Start a new server session for tool choice examples server_process_tool_choice, port_tool_choice = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port_tool_choice}") # Initialize client for tool choice examples client_tool_choice = OpenAI( api_key="None", base_url=f"http://0.0.0.0:{port_tool_choice}/v1" ) model_name_tool_choice = client_tool_choice.models.list().data[0].id # Example with tool_choice="required" - forces the model to call a tool messages_required = [ {"role": "user", "content": "Hello, what is the capital of France?"} ] # Define tools tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The city to find the weather for, e.g. 'San Francisco'", }, "unit": { "type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"], }, }, "required": ["city", "unit"], }, }, } ] response_required = client_tool_choice.chat.completions.create( model=model_name_tool_choice, messages=messages_required, temperature=0, max_tokens=1024, tools=tools, tool_choice="required", # Force the model to call a tool ) print_highlight("Response with tool_choice='required':") print("Content:", response_required.choices[0].message.content) print("Tool calls:", response_required.choices[0].message.tool_calls) ``` ### Example: Specific Function Choice ```python Example theme={null} # Example with specific function choice - forces the model to call a specific function messages_specific = [ {"role": "user", "content": "What are the most attactive places in France?"} ] response_specific = client_tool_choice.chat.completions.create( model=model_name_tool_choice, messages=messages_specific, temperature=0, max_tokens=1024, tools=tools, tool_choice={ "type": "function", "function": {"name": "get_current_weather"}, }, # Force the model to call the specific get_current_weather function ) print_highlight("Response with specific function choice:") print("Content:", response_specific.choices[0].message.content) print("Tool calls:", response_specific.choices[0].message.tool_calls) if response_specific.choices[0].message.tool_calls: tool_call = response_specific.choices[0].message.tool_calls[0] print_highlight(f"Called function: {tool_call.function.name}") print_highlight(f"Arguments: {tool_call.function.arguments}") ``` ```python Example theme={null} terminate_process(server_process_tool_choice) ``` ## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4) Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a "pythonic" tool call format, where the model outputs function calls as Python code, e.g.: ```python Example theme={null} [get_current_weather(city="San Francisco", state="CA", unit="celsius")] ``` * The output is a Python list of function calls, with arguments as Python literals (not JSON). * Multiple tool calls can be returned in the same list: ```python Example theme={null} [get_current_weather(city="San Francisco", state="CA", unit="celsius"), get_current_weather(city="New York", state="NY", unit="fahrenheit")] ``` For more information, refer to Meta’s documentation on [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message). Note that this feature is still under development on Blackwell. ### How to enable * Launch the server with `--tool-call-parser pythonic` * You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`). This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results. #### Forcing Pythonic Tool Call Output Without a Chat Template If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need: ```python Example theme={null} import openai server_process, port = launch_server_cmd( " python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1 --log-level warning" # llama-3.2-1b-instruct ) wait_for_server(f"http://localhost:{port}") tools = [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The name of the city or location.", } }, "required": ["location"], }, }, }, { "type": "function", "function": { "name": "get_tourist_attractions", "description": "Get a list of top tourist attractions for a given city.", "parameters": { "type": "object", "properties": { "city": { "type": "string", "description": "The name of the city to find attractions for.", } }, "required": ["city"], }, }, }, ] def get_messages(): return [ { "role": "system", "content": ( "You are a travel assistant. " "When asked to call functions, ALWAYS respond ONLY with a python list of function calls, " "using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. " "Do NOT use JSON, do NOT use variables, do NOT use any other format. " "Here is an example:\n" '[get_weather(location="Paris"), get_tourist_attractions(city="Paris")]' ), }, { "role": "user", "content": ( "I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? " "Propose parallel tool calls at once, using the python list of function calls format as shown above." ), }, ] messages = get_messages() client = openai.Client(base_url=f"http://localhost:{port}/v1", api_key="xxxxxx") model_name = client.models.list().data[0].id response_non_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.9, stream=False, # Non-streaming tools=tools, ) print_highlight("Non-stream response:") print_highlight(response_non_stream) response_stream = client.chat.completions.create( model=model_name, messages=messages, temperature=0, top_p=0.9, stream=True, tools=tools, ) texts = "" tool_calls = [] name = "" arguments = "" for chunk in response_stream: if chunk.choices[0].delta.content: texts += chunk.choices[0].delta.content if chunk.choices[0].delta.tool_calls: tool_calls.append(chunk.choices[0].delta.tool_calls[0]) print_highlight("Streaming Response:") print_highlight("==== Text ====") print_highlight(texts) print_highlight("==== Tool Call ====") for tool_call in tool_calls: print_highlight(tool_call) terminate_process(server_process) ``` > **Note:** > The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template. ## How to support a new model? 1. Update the TOOLS\_TAG\_LIST in sglang/srt/function\_call\_parser.py with the model’s tool tags. Currently supported tags include: ```text Output theme={null} TOOLS_TAG_LIST = [ “<|plugin|>“, ““, “<|python_tag|>“, “[TOOL_CALLS]” ] ``` 2. Create a new detector class in sglang/srt/function\_call\_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example: ```text Output theme={null} class NewModelDetector(BaseFormatDetector): ``` 3. Add the new detector to the MultiFormatParser class that manages all the format detectors. # Query VLM with Offline Engine Source: https://docs.sglang.io/docs/advanced_features/vlm_query This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches: 1. **Basic Call**: Directly pass images and text. 2. **Processor Output**: Use HuggingFace processor for data preprocessing. 3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency. ## Understanding the Three Input Formats SGLang supports three ways to pass visual data, each optimized for different scenarios: ### 1. **Raw Images** - Simplest approach * Pass PIL Images, file paths, URLs, or base64 strings directly * SGLang handles all preprocessing automatically * Best for: Quick prototyping, simple applications ### 2. **Processor Output** - For custom preprocessing * Pre-process images with HuggingFace processor * Pass the complete processor output dict with `format: "processor_output"` * Best for: Custom image transformations, integration with existing pipelines * Requirement: Must use `input_ids` instead of text prompt ### 3. **Precomputed Embeddings** - For maximum performance * Pre-calculate visual embeddings using the vision encoder * Pass embeddings with `format: "precomputed_embedding"` * Best for: Repeated queries on same images, caching, high-throughput serving * Performance gain: Avoids redundant vision encoder computation (30-50% speedup) **Key Rule**: Within a single request, use only one format for all images. Don't mix formats. The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models. ## Querying Qwen2.5-VL Model ```python Example theme={null} import nest_asyncio nest_asyncio.apply() import sglang.test.doc_patch # noqa: F401 model_path = "Qwen/Qwen2.5-VL-3B-Instruct" chat_template = "qwen2-vl" example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" ``` ```python Example theme={null} from io import BytesIO import requests from PIL import Image from sglang.srt.parser.conversation import chat_templates image = Image.open(BytesIO(requests.get(example_image_url).content)) conv = chat_templates[chat_template].copy() conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?") conv.append_message(conv.roles[1], "") conv.image_data = [image] print("Generated prompt text:") print(conv.get_prompt()) print(f"\nImage size: {image.size}") image ``` ### Basic Offline Engine API Call ```python Example theme={null} from sglang import Engine llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning") ``` ```python Example theme={null} out = llm.generate(prompt=conv.get_prompt(), image_data=[image]) print("Model response:") print(out["text"]) ``` ### Call with Processor Output Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`. ```python Example theme={null} from transformers import AutoProcessor processor = AutoProcessor.from_pretrained(model_path) processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) out = llm.generate( input_ids=processor_output["input_ids"][0].detach().cpu().tolist(), image_data=[dict(processor_output, format="processor_output")], ) print("Response using processor output:") print(out["text"]) ``` ### Call with Precomputed Embeddings You can pre-calculate image features to avoid repeated visual encoding processes. ```python Example theme={null} from transformers import AutoProcessor from transformers import Qwen2_5_VLForConditionalGeneration processor = AutoProcessor.from_pretrained(model_path) model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval() vision = model.model.visual.cuda() ``` ```python Example theme={null} processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) input_ids = processor_output["input_ids"][0].detach().cpu().tolist() precomputed_embeddings = vision( processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda() ) precomputed_embeddings = precomputed_embeddings.pooler_output multi_modal_item = dict( processor_output, format="precomputed_embedding", feature=precomputed_embeddings, ) out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item]) print("Response using precomputed embeddings:") print(out["text"]) llm.shutdown() ``` ## Querying Llama 4 Vision Model ```python Example theme={null} model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct" chat_template = "llama-4" from io import BytesIO import requests from PIL import Image from sglang.srt.parser.conversation import chat_templates # Download the same example image image = Image.open(BytesIO(requests.get(example_image_url).content)) conv = chat_templates[chat_template].copy() conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?") conv.append_message(conv.roles[1], "") conv.image_data = [image] print("Llama 4 generated prompt text:") print(conv.get_prompt()) print(f"Image size: {image.size}") image ``` ### Llama 4 Basic Call Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp\_size=4) and larger context length. ```python Example theme={null} llm = Engine( model_path=model_path, enable_multimodal=True, attention_backend="fa3", tp_size=4, context_length=65536, ) out = llm.generate(prompt=conv.get_prompt(), image_data=[image]) print("Llama 4 response:") print(out["text"]) ``` ### Call with Processor Output Using HuggingFace processor to preprocess data can reduce computational overhead during inference. ```python Example theme={null} from transformers import AutoProcessor processor = AutoProcessor.from_pretrained(model_path) processor_output = processor( images=[image], text=conv.get_prompt(), return_tensors="pt" ) out = llm.generate( input_ids=processor_output["input_ids"][0].detach().cpu().tolist(), image_data=[dict(processor_output, format="processor_output")], ) print("Response using processor output:") print(out) ``` ### Call with Precomputed Embeddings ```python Example theme={null} from transformers import AutoProcessor from transformers import Llama4ForConditionalGeneration processor = AutoProcessor.from_pretrained(model_path) model = Llama4ForConditionalGeneration.from_pretrained( model_path, torch_dtype="auto" ).eval() vision = model.vision_model.cuda() multi_modal_projector = model.multi_modal_projector.cuda() print(f'Image pixel values shape: {processor_output["pixel_values"].shape}') input_ids = processor_output["input_ids"][0].detach().cpu().tolist() # Process image through vision encoder image_outputs = vision( processor_output["pixel_values"].to("cuda"), aspect_ratio_ids=processor_output["aspect_ratio_ids"].to("cuda"), aspect_ratio_mask=processor_output["aspect_ratio_mask"].to("cuda"), output_hidden_states=False ) image_features = image_outputs.last_hidden_state # Flatten image features and pass through multimodal projector vision_flat = image_features.view(-1, image_features.size(-1)) precomputed_embeddings = multi_modal_projector(vision_flat) # Build precomputed embedding data item mm_item = dict( processor_output, format="precomputed_embedding", feature=precomputed_embeddings ) # Use precomputed embeddings for efficient inference out = llm.generate(input_ids=input_ids, image_data=[mm_item]) print("Llama 4 precomputed embedding response:") print(out["text"]) ``` # Anthropic-Compatible API Source: https://docs.sglang.io/docs/basic_usage/anthropic_api Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning. SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a self-hosted SGLang server without changes. A complete reference for the API is available in the [Anthropic API Reference](https://docs.anthropic.com/en/api/messages). The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it. It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route. This tutorial covers: * `POST /v1/messages` (non-streaming and streaming) * `POST /v1/messages/count_tokens` * Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is required for good prefix-cache reuse. ## Launch A Server Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint is registered automatically — no extra flag is required beyond the usual server launch. The example below is a single-node GLM-5.2-FP8 config; see the [GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands across hardware and quantizations. ```bash Command theme={null} sglang serve \ --model-path zai-org/GLM-5.2-FP8 \ --tp 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 5 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 6 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --host 0.0.0.0 \ --port 30000 ``` * **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is used here because its reasoning + tool-use output is where Claude Code integration shines, but any model works. * **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the `ANTHROPIC_DEFAULT_*_MODEL` env vars below. * **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use` blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back as raw text, and Claude Code cannot execute them. * **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it. ## Send A Message ### Non-Streaming Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends `/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix. ```python Example theme={null} from anthropic import Anthropic client = Anthropic( base_url="http://127.0.0.1:30000", api_key="EMPTY", # SGLang does not require a real key by default ) message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, messages=[{"role": "user", "content": "List 3 countries and their capitals."}], ) # A reasoning model may emit a `thinking` block before the `text` block — # pick the text block rather than assuming content[0]. print(next(b.text for b in message.content if b.type == "text")) ``` **Example Output:** ```text Output theme={null} Here are 3 countries and their capitals: 1. **France** - Paris 2. **Japan** - Tokyo 3. **Brazil** - Brasília ``` ### Streaming Set `stream=True` to receive Server-Sent Events as they are produced. ```python Example theme={null} with client.messages.stream( model="zai-org/GLM-5.2-FP8", max_tokens=512, messages=[{"role": "user", "content": "Say this is a test"}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) ``` **Example Output:** ```text Output theme={null} This is a test. ``` ### System Prompt The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic API shape: ```python Example theme={null} message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, system="You are a helpful assistant that answers concisely.", messages=[{"role": "user", "content": "What is the capital of France?"}], ) print(next(b.text for b in message.content if b.type == "text")) ``` **Example Output:** ```text Output theme={null} The capital of France is Paris. ``` ### Tool Use Tool definitions follow the Anthropic `tools` schema. When the server is launched with a `--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks: ```python Example theme={null} message = client.messages.create( model="zai-org/GLM-5.2-FP8", max_tokens=512, tools=[ { "name": "get_weather", "description": "Get the weather for a city", "input_schema": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, } ], messages=[{"role": "user", "content": "What is the weather in Paris?"}], ) print(message.stop_reason) print([b for b in message.content if b.type == "tool_use"]) ``` **Example Output:** ```text Output theme={null} tool_use [ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})] ``` ### Counting Tokens `POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and multi-turn history are all accounted for. ```python Example theme={null} resp = client.messages.count_tokens( model="zai-org/GLM-5.2-FP8", messages=[{"role": "user", "content": "Hello, world"}], ) print(resp.input_tokens) ``` **Example Output:** ```text Output theme={null} 15 ``` ## Using Claude Code Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it. With the server already running on `:30000`, export the full set and launch `claude`: ```bash Command theme={null} export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" export ANTHROPIC_AUTH_TOKEN="dummy" # required by Claude Code; any non-empty string works export API_TIMEOUT_MS="3000000" # long timeout — reasoning + 1M-context turns are slow export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000" # let auto-compact use the full 1M window export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 # drop autoupdater/telemetry/error-reporting noise export CLAUDE_CODE_ATTRIBUTION_HEADER=0 # required for prefix-cache reuse — see below export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]" # [1m] suffix enables Claude Code's 1M-context beta claude ``` Each var matters: * **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API. * **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value when launched without `--api-key`. * **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely exceed the default timeout. * **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier. SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a client-side hint that enables Claude Code's 1M-context beta (without it, context is capped). * **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window instead of the default, keeping long sessions alive. Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key — they apply to all Claude Code sessions: ```json theme={null} { "env": { "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000", "ANTHROPIC_AUTH_TOKEN": "dummy", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "CLAUDE_CODE_ATTRIBUTION_HEADER": "0", "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]", "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]", "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]" } } ``` ### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse **Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it, multi-turn conversations re-prefill the whole history every turn. Claude Code prepends a per-request attribution block to the start of the system prompt, of the form `x-anthropic-billing-header: cc_version=.; cc_entrypoint=...; cch=;`. The per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse the short prefix before that hash and re-prefills the system prompt plus the entire conversation history on every turn. Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt. This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)). `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers autoupdater/telemetry/error reporting. The attribution header is a separate code path; use `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it. ## Troubleshooting **Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL` matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable and not behind a proxy that blocks the connection. **`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and serves whatever model was loaded at startup, so a 404 usually means the request did not reach the `/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and that the server finished loading. **Tool calls not working / returned as raw text** — Launch the server with the correct `--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute them. **Slow / re-prefills the whole history every turn** — You are missing `CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt defeats radix prefix-cache reuse; see the section above. **Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend). ## Parameters The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the [Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list. Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models, `enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for the reasoning-parser / chat-template mapping. # Amazon SageMaker AI Source: https://docs.sglang.io/docs/basic_usage/aws_sagemaker Deploy SGLang on Amazon SageMaker AI endpoints using the AWS Deep Learning Container. Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoints using the [AWS Deep Learning Container (DLC)](https://aws.github.io/deep-learning-containers/sglang/) for SGLang. The SageMaker image variant accepts model configuration via environment variables and serves on port 8080. This guide uses the pre-built DLC image. To build and deploy your own container instead, see [Method 7: Run on AWS SageMaker](/docs/get-started/install#more-3) in the installation guide. ## Container image AWS publishes pre-built, security-patched SGLang DLCs. The SageMaker GPU image is available from the Amazon ECR registry (account `763104351884`) in each supported region. For example, in `us-west-2`: ```text theme={null} 763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0 ``` For the full list of image tags, see the [Available DLC Images](https://aws.github.io/deep-learning-containers/reference/available_images/) reference, and for region-specific account IDs and supported regions, see [Region Availability](https://aws.github.io/deep-learning-containers/reference/region_availability/). ## Specifying the model The SageMaker image resolves the model in this order: 1. **`SM_SGLANG_MODEL_PATH` environment variable** — explicit Hugging Face ID or path. 2. **`/opt/ml/model`** — when SageMaker mounts model artifacts via `ModelDataUrl` or `ModelDataSource`, the entrypoint uses this path by default. For gated models, also pass `HF_TOKEN`. Any `SM_SGLANG_*` environment variable is converted to a `--` SGLang server argument (for example, `SM_SGLANG_CONTEXT_LENGTH=4096` becomes `--context-length 4096`). ## Deploy with the SageMaker Python SDK ```python theme={null} from sagemaker.model import Model from sagemaker.predictor import Predictor from sagemaker.serializers import JSONSerializer model = Model( image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0", role="arn:aws:iam:::role/", predictor_cls=Predictor, env={"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"}, ) predictor = model.deploy( instance_type="ml.g5.2xlarge", initial_instance_count=1, inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1", serializer=JSONSerializer(), ) response = predictor.predict({ "model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "What is deep learning?"}], "max_tokens": 256, }) print(response) # Cleanup predictor.delete_model() predictor.delete_endpoint(delete_endpoint_config=True) ``` ## Deploy with Boto3 ```python theme={null} import json import boto3 sm = boto3.client("sagemaker") smrt = boto3.client("sagemaker-runtime") sm.create_model( ModelName="sglang-model", PrimaryContainer={ "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0", "Environment": {"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"}, }, ExecutionRoleArn="arn:aws:iam:::role/", ) sm.create_endpoint_config( EndpointConfigName="sglang-config", ProductionVariants=[{ "VariantName": "default", "ModelName": "sglang-model", "InstanceType": "ml.g5.2xlarge", "InitialInstanceCount": 1, "InferenceAmiVersion": "al2023-ami-sagemaker-inference-gpu-4-1", }], ) sm.create_endpoint(EndpointName="sglang-endpoint", EndpointConfigName="sglang-config") sm.get_waiter("endpoint_in_service").wait(EndpointName="sglang-endpoint") resp = smrt.invoke_endpoint( EndpointName="sglang-endpoint", ContentType="application/json", Body=json.dumps({ "model": "openai/gpt-oss-20b", "messages": [{"role": "user", "content": "What is deep learning?"}], "max_tokens": 256, }), ) print(json.loads(resp["Body"].read())) # Cleanup sm.delete_endpoint(EndpointName="sglang-endpoint") sm.delete_endpoint_config(EndpointConfigName="sglang-config") sm.delete_model(ModelName="sglang-model") ``` ## Model artifacts When `ModelDataUrl` (or `ModelDataSource`) points to a tarball or S3 prefix, SageMaker mounts the contents at `/opt/ml/model`. The entrypoint defaults `--model-path` to that location, so `SM_SGLANG_MODEL_PATH` can be omitted: ```text theme={null} model.tar.gz ├── config.json # standard model files (Hugging Face layout) ├── tokenizer.json └── *.safetensors ``` ## Notes * GPU deployments require `inference_ami_version` — the default SageMaker host AMI has incompatible NVIDIA drivers for CUDA 13 images. See the [ProductionVariant API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html) for valid values. * The endpoint exposes an OpenAI-compatible API, so the request body matches the SGLang server's `/v1/chat/completions` schema. # SGLang Native APIs Source: https://docs.sglang.io/docs/basic_usage/native_api Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs: * `/generate` (text generation model) * `/get_model_info` * `/server_info` * `/health` * `/health_generate` * `/flush_cache` * `/update_weights` * `/encode`(embedding model) * `/v1/rerank`(cross encoder rerank model) * `/v1/score`(decoder-only scoring) * `/classify`(reward model) * `/start_expert_distribution_record` * `/stop_expert_distribution_record` * `/dump_expert_distribution_record` * `/tokenize` * `/detokenize` * A full list of these APIs can be found at [http\_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py) We mainly use `requests` to test these APIs in the following examples. You can also use `curl`. ## Launch A Server ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) ``` ## Generate (text generation model) Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params). ```python Example theme={null} import requests url = f"http://localhost:{port}/generate" data = {"text": "What is the capital of France?"} response = requests.post(url, json=data) print_highlight(response.json()) ``` ## Get Model Info Get the information of the model. * `model_path`: The path/name of the model. * `is_generation`: Whether the model is used as generation model or embedding model. * `tokenizer_path`: The path/name of the tokenizer. * `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args. * `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters. * `has_image_understanding`: Whether the model has image-understanding capability. * `has_audio_understanding`: Whether the model has audio-understanding capability. * `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama"). * `architectures`: The model architectures from the HuggingFace config (e.g., \["Qwen2ForCausalLM"]). * `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract. ```python Example theme={null} url = f"http://localhost:{port}/get_model_info" response = requests.get(url) response_json = response.json() print_highlight(response_json) assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct" assert response_json["is_generation"] is True assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct" assert response_json["preferred_sampling_params"] is None assert response_json.keys() == { "model_path", "is_generation", "tokenizer_path", "preferred_sampling_params", "weight_version", "has_image_understanding", "has_audio_understanding", "model_type", "architectures", "embedding", } ``` ## Get Server Info Gets the server information including CLI arguments, token limits, and memory pool sizes. * Note: `get_server_info` merges the following deprecated endpoints: * `get_server_args` * `get_memory_pool_size` * `get_max_total_num_tokens` ```python Example theme={null} url = f"http://localhost:{port}/server_info" response = requests.get(url) print_highlight(response.text) ``` ## Health Check * `/health`: Check the health of the server. * `/health_generate`: Check the health of the server by generating one token. ```python Example theme={null} url = f"http://localhost:{port}/health_generate" response = requests.get(url) print_highlight(response.text) ``` ```python Example theme={null} url = f"http://localhost:{port}/health" response = requests.get(url) print_highlight(response.text) ``` ## Flush Cache Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API. Parameters: * `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors. ```bash Command theme={null} # With timeout (wait up to 30s for idle state) curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30" ``` ```python Example theme={null} url = f"http://localhost:{port}/flush_cache" response = requests.post(url) print_highlight(response.text) ``` ## Update Weights From Disk Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size. SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk). ```python Example theme={null} # successful update with same architecture and size url = f"http://localhost:{port}/update_weights_from_disk" data = {"model_path": "qwen/qwen2.5-0.5b-instruct"} response = requests.post(url, json=data) print_highlight(response.text) assert response.json()["success"] is True assert response.json()["message"] == "Succeeded to update model weights." ``` ```python Example theme={null} # failed update with different parameter size or wrong name url = f"http://localhost:{port}/update_weights_from_disk" data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"} response = requests.post(url, json=data) response_json = response.json() print_highlight(response_json) assert response_json["success"] is False assert response_json["message"] == ( "Failed to get weights iterator: " "qwen/qwen2.5-0.5b-instruct-wrong" " (repository not found)." ) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Encode (embedding model) Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models. Therefore, we launch a new server to server an embedding model. ```python Example theme={null} embedding_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \ --host 0.0.0.0 --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=embedding_process) ``` ```python Example theme={null} # successful encode for embedding model url = f"http://localhost:{port}/encode" data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"} response = requests.post(url, json=data) response_json = response.json() print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}") ``` ```python Example theme={null} terminate_process(embedding_process) ``` ## v1/rerank (cross encoder rerank model) Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`. ```python Example theme={null} reranker_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \ --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=reranker_process) ``` ```python Example theme={null} # compute rerank scores for query and documents url = f"http://localhost:{port}/v1/rerank" data = { "model": "BAAI/bge-reranker-v2-m3", "query": "what is panda?", "documents": [ "hi", "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.", ], } response = requests.post(url, json=data) response_json = response.json() for item in response_json: print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'") ``` ```python Example theme={null} terminate_process(reranker_process) ``` ## v1/score (decoder-only scoring) Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities. Parameters: * `query`: Query text * `items`: Item text(s) to score * `label_token_ids`: Token IDs to compute probabilities for * `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False) * `item_first`: Whether items come first in concatenation order (default: False) * `model`: Model name The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`. ```python Example theme={null} score_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \ --host 0.0.0.0 --log-level warning """) wait_for_server(f"http://localhost:{port}", process=score_process) ``` ```python Example theme={null} # Score the probability of different completions given a query query = "The capital of France is" items = ["Paris", "London", "Berlin"] url = f"http://localhost:{port}/v1/score" data = { "model": "qwen/qwen2.5-0.5b-instruct", "query": query, "items": items, "label_token_ids": [9454, 2753], # e.g. "Yes" and "No" token ids "apply_softmax": True, # Normalize probabilities to sum to 1 } response = requests.post(url, json=data) response_json = response.json() # Display scores for each item for item, scores in zip(items, response_json["scores"]): print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}") ``` ```python Example theme={null} terminate_process(score_process) ``` ## Classify (reward model) SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations. ```python Example theme={null} # Note that SGLang now treats embedding models and reward models as the same type of models. # This will be updated in the future. reward_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning """) wait_for_server(f"http://localhost:{port}", process=reward_process) ``` ```python Example theme={null} from transformers import AutoTokenizer PROMPT = ( "What is the range of the numeric output of a sigmoid node in a neural network?" ) RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1." RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1." CONVS = [ [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}], [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}], ] tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2") prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False) url = f"http://localhost:{port}/classify" data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts} responses = requests.post(url, json=data).json() for response in responses: print_highlight(f"reward: {response['embedding'][0]}") ``` ```python Example theme={null} terminate_process(reward_process) ``` ## Capture expert selection distribution in MoE models SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization. *Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.* ```python Example theme={null} expert_record_server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=expert_record_server_process) ``` ```python Example theme={null} response = requests.post(f"http://localhost:{port}/start_expert_distribution_record") print_highlight(response) url = f"http://localhost:{port}/generate" data = {"text": "What is the capital of France?"} response = requests.post(url, json=data) print_highlight(response.json()) response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record") print_highlight(response) response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record") print_highlight(response) ``` ```python Example theme={null} terminate_process(expert_record_server_process) ``` ## Tokenize/Detokenize Example (Round Trip) This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization. ```python Example theme={null} tokenizer_free_server_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct """) wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process) ``` ```python Example theme={null} import requests from sglang.utils import print_highlight base_url = f"http://localhost:{port}" tokenize_url = f"{base_url}/tokenize" detokenize_url = f"{base_url}/detokenize" model_name = "qwen/qwen2.5-0.5b-instruct" input_text = "SGLang provides efficient tokenization endpoints." print_highlight(f"Original Input Text:\n'{input_text}'") # --- tokenize the input text --- tokenize_payload = { "model": model_name, "prompt": input_text, "add_special_tokens": False, } try: tokenize_response = requests.post(tokenize_url, json=tokenize_payload) tokenize_response.raise_for_status() tokenization_result = tokenize_response.json() token_ids = tokenization_result.get("tokens") if not token_ids: raise ValueError("Tokenization returned empty tokens.") print_highlight(f"\nTokenized Output (IDs):\n{token_ids}") print_highlight(f"Token Count: {tokenization_result.get('count')}") print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}") # --- detokenize the obtained token IDs --- detokenize_payload = { "model": model_name, "tokens": token_ids, "skip_special_tokens": True, } detokenize_response = requests.post(detokenize_url, json=detokenize_payload) detokenize_response.raise_for_status() detokenization_result = detokenize_response.json() reconstructed_text = detokenization_result.get("text") print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'") if input_text == reconstructed_text: print_highlight( "\nRound Trip Successful: Original and reconstructed text match." ) else: print_highlight( "\nRound Trip Mismatch: Original and reconstructed text differ." ) except requests.exceptions.RequestException as e: print_highlight(f"\nHTTP Request Error: {e}") except Exception as e: print_highlight(f"\nAn error occurred: {e}") ``` ```python Example theme={null} terminate_process(tokenizer_free_server_process) ``` # Offline Engine API Source: https://docs.sglang.io/docs/basic_usage/offline_engine_api SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases: * Offline Batch Inference * Custom Server on Top of the Engine This document focuses on the offline batch inference, demonstrating four different inference modes: * Non-streaming synchronous generation * Streaming synchronous generation * Non-streaming asynchronous generation * Streaming asynchronous generation Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom\_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py). ## Nest Asyncio Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code: ```python Example theme={null} import nest_asyncio nest_asyncio.apply() ``` ## Advanced Usage The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states). Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases. ## Offline Batch Inference SGLang offline engine supports batch inference with efficient scheduling. ```python Example theme={null} # launch the offline engine import asyncio import sglang as sgl import sglang.test.doc_patch from sglang.utils import async_stream_and_merge, stream_and_merge llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct") ``` ### Non-streaming Synchronous Generation ```python Example theme={null} prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print("===============================") print(f"Prompt: {prompt}\nGenerated text: {output['text']}") ``` ### Streaming Synchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = { "temperature": 0.2, "top_p": 0.9, } print("\n=== Testing synchronous streaming generation with overlap removal ===\n") for prompt in prompts: print(f"Prompt: {prompt}") merged_output = stream_and_merge(llm, prompt, sampling_params) print("Generated text:", merged_output) print() ``` ### Non-streaming Asynchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} print("\n=== Testing asynchronous batch generation ===") async def main(): outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"\nPrompt: {prompt}") print(f"Generated text: {output['text']}") asyncio.run(main()) ``` ### Streaming Asynchronous Generation ```python Example theme={null} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} print("\n=== Testing asynchronous streaming generation (no repeats) ===") async def main(): for prompt in prompts: print(f"\nPrompt: {prompt}") print("Generated text: ", end="", flush=True) # Replace direct calls to async_generate with our custom overlap-aware version async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params): print(cleaned_chunk, end="", flush=True) print() # New line after each prompt asyncio.run(main()) ``` ```python Example theme={null} llm.shutdown() ``` # Ollama-Compatible API Source: https://docs.sglang.io/docs/basic_usage/ollama_api SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend. ## Prerequisites ```bash Command theme={null} # Install the Ollama Python library (for Python client usage) pip install ollama ``` You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client. ## Endpoints
Endpoint Method Description
`/` GET, HEAD Health check for Ollama CLI
`/api/tags` GET List available models
`/api/chat` POST Chat completions (streaming & non-streaming)
`/api/generate` POST Text generation (streaming & non-streaming)
`/api/show` POST Model information
## Quick Start ### 1. Launch SGLang Server ```bash Command theme={null} python -m sglang.launch_server \ --model Qwen/Qwen2.5-1.5B-Instruct \ --port 30001 \ --host 0.0.0.0 ``` The model name used with `ollama run` must match exactly what you passed to `--model`. ### 2. Use Ollama CLI ```bash Command theme={null} # List available models OLLAMA_HOST=http://localhost:30001 ollama list # Interactive chat OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct" ``` If connecting to a remote server behind a firewall: ```bash Command theme={null} # SSH tunnel ssh -L 30001:localhost:30001 user@gpu-server -N & # Then use Ollama CLI as above OLLAMA_HOST=http://localhost:30001 ollama list ``` ### 3. Use Ollama Python Library ```python Example theme={null} import ollama client = ollama.Client(host='http://localhost:30001') # Non-streaming response = client.chat( model='Qwen/Qwen2.5-1.5B-Instruct', messages=[{'role': 'user', 'content': 'Hello!'}] ) print(response['message']['content']) # Streaming stream = client.chat( model='Qwen/Qwen2.5-1.5B-Instruct', messages=[{'role': 'user', 'content': 'Tell me a story'}], stream=True ) for chunk in stream: print(chunk['message']['content'], end='', flush=True) ``` ## Smart Router For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README.md). ## Summary
Component Purpose
**Ollama API** Familiar CLI/API that developers already know
**SGLang Backend** High-performance inference engine
**Smart Router** Intelligent routing - fast local for simple tasks, powerful remote for complex tasks
# OpenAI-Compatible APIs Source: https://docs.sglang.io/docs/basic_usage/openai_api Documentation for OpenAI-Compatible APIs * [Openai Api Completions](./openai_api_completions) * [Openai Api Vision](./openai_api_vision) * [Openai Api Embeddings](./openai_api_embeddings) For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api). # OpenAI APIs - Completions Source: https://docs.sglang.io/docs/basic_usage/openai_api_completions SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference). This tutorial covers the following popular APIs: * `chat/completions` * `completions` Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models. ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process server_process, port = launch_server_cmd( "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}") print(f"Server started on http://localhost:{port}") ``` ## Chat Completions ### Usage The server fully implements the OpenAI API. It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available. You can also specify a custom chat template with `--chat-template` when launching the server. ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print_highlight(f"Response: {response}") ``` ### Model Thinking/Reasoning Support Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers. #### Supported Models and Configuration
Model Family Chat Template Parameter Reasoning Parser Notes
DeepSeek-R1 (R1, R1-0528, R1-Distill) `enable_thinking` `--reasoning-parser deepseek-r1` Standard reasoning models
DeepSeek-V3.1 `thinking` `--reasoning-parser deepseek-v3` Hybrid model (thinking/non-thinking modes)
Qwen3 (standard) `enable_thinking` `--reasoning-parser qwen3` Hybrid model (thinking/non-thinking modes)
Qwen3-Thinking N/A (always enabled) `--reasoning-parser qwen3-thinking` Always generates reasoning
Kimi N/A (always enabled) `--reasoning-parser kimi` Kimi thinking models
Gpt-Oss N/A (always enabled) `--reasoning-parser gpt-oss` Gpt-Oss thinking models
#### Basic Usage To enable reasoning output, you need to: 1. Launch the server with the appropriate reasoning parser 2. Set the model-specific parameter in `chat_template_kwargs` 3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`) **Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content. #### Example: Qwen3 Models ```python Example theme={null} # Launch server: # python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3 from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url=f"http://127.0.0.1:30000/v1", ) model = "Qwen/Qwen3-4B" messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}] response = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": {"enable_thinking": True}, "separate_reasoning": True } ) print("Reasoning:", response.choices[0].message.reasoning_content) print("-"*100) print("Answer:", response.choices[0].message.content) ``` **ExampleOutput:** ```text Output theme={null} Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down. Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y. ... Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right. ---------------------------------------------------------------------------------------------------- Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown: 1. **S-T-R-A-W-B-E-R-R-Y** - The **third letter** is 'R'. - The **eighth and ninth letters** are also 'R's. Thus, the total count is **3**. **Answer:** 3. ``` Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter. #### Logit Bias Support SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where: * **Positive values** (0 to 100) increase the likelihood of the token being selected * **Negative values** (-100 to 0) decrease the likelihood of the token being selected * **-100** effectively prevents the token from being generated The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats). #### Getting Token IDs To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs: ```python Example theme={null} # Get tokenizer to find token IDs import tiktoken # For OpenAI models, use the appropriate encoding tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo") # or your model # Get token IDs for specific words word = "sunny" token_ids = tokenizer.encode(word) print(f"Token IDs for '{word}': {token_ids}") # For SGLang models, you can access the tokenizer through the client # and get token IDs for bias ``` **Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words. #### Example: DeepSeek-V3 Models DeepSeek-V3 models support thinking mode through the `thinking` parameter: ```python Example theme={null} # Launch server: # python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8 --reasoning-parser deepseek-v3 from openai import OpenAI client = OpenAI( api_key="EMPTY", base_url=f"http://127.0.0.1:30000/v1", ) model = "deepseek-ai/DeepSeek-V3.1" messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}] response = client.chat.completions.create( model=model, messages=messages, extra_body={ "chat_template_kwargs": {"thinking": True}, "separate_reasoning": True } ) print("Reasoning:", response.choices[0].message.reasoning_content) print("-"*100) print("Answer:", response.choices[0].message.content) ``` **Example Output:** ```text Output theme={null} Reasoning: First, the question is: "How many r's are in 'strawberry'?" I need to count the number of times the letter 'r' appears in the word "strawberry". Let me write out the word: S-T-R-A-W-B-E-R-R-Y. Now, I'll go through each letter and count the 'r's. ... So, I have three 'r's in "strawberry". I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct. Therefore, the answer should be 3. ---------------------------------------------------------------------------------------------------- Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity: - The word is spelled: S-T-R-A-W-B-E-R-R-Y - The "r" appears at the 3rd, 8th, and 9th positions. ``` DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output. ```python Example theme={null} # Example with logit_bias parameter # Note: You need to get the actual token IDs from your tokenizer # For demonstration, we'll use some example token IDs response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "Complete this sentence: The weather today is"} ], temperature=0.7, max_tokens=20, logit_bias={ "12345": 50, # Increase likelihood of token ID 12345 "67890": -50, # Decrease likelihood of token ID 67890 "11111": 25, # Slightly increase likelihood of token ID 11111 }, ) print_highlight(f"Response with logit bias: {response.choices[0].message.content}") ``` ### Parameters The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details. SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor. ```python Example theme={null} response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ { "role": "system", "content": "You are a knowledgeable historian who provides concise responses.", }, {"role": "user", "content": "Tell me about ancient Rome"}, { "role": "assistant", "content": "Ancient Rome was a civilization centered in Italy.", }, {"role": "user", "content": "What were their major achievements?"}, ], temperature=0.3, # Lower temperature for more focused responses max_tokens=128, # Reasonable length for a concise response top_p=0.95, # Slightly higher for better fluency presence_penalty=0.2, # Mild penalty to avoid repetition frequency_penalty=0.2, # Mild penalty for more natural language n=1, # Single response is usually more stable seed=42, # Keep for reproducibility ) print_highlight(response.choices[0].message.content) ``` Streaming mode is also supported. #### Logit Bias Support The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above. ```python Example theme={null} stream = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "Say this is a test"}], stream=True, ) for chunk in stream: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="") ``` #### Returning Routed Experts (MoE Models) For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. By default, `routed_experts` is returned in the response-level `sglext` object as base64-encoded int32 expert IDs in a flattened array with logical shape `[num_tokens, num_layers, top_k]`. For non-streaming chat responses with `return_meta_info: true`, `routed_experts` is returned only in each choice's `meta_info` object and omitted from response-level `sglext` to avoid duplicating the routing data. Other requested extension fields, such as `cached_tokens_details` and `spec_tokens_details`, remain in response-level `sglext`. By default, routed experts cover `[0, seqlen - 1)`, the full available sequence, because RL workflows need them for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that causes bottlenecks. ```python Example theme={null} # Example with logit_bias parameter for completions API # Note: You need to get the actual token IDs from your tokenizer # For demonstration, we'll use some example token IDs response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="The best programming language for AI is", temperature=0.7, max_tokens=20, logit_bias={ "12345": 75, # Strongly favor token ID 12345 "67890": -100, # Completely avoid token ID 67890 "11111": -25, # Slightly discourage token ID 11111 }, ) print_highlight(f"Response with logit bias: {response.choices[0].text}") ``` ## Completions ### Usage Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates. ```python Example theme={null} response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="List 3 countries and their capitals.", temperature=0, max_tokens=64, n=1, stop=None, ) print_highlight(f"Response: {response}") ``` ### Parameters The completions API accepts OpenAI Completions API's parameters. Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details. Here is an example of a detailed completions request: ```python Example theme={null} response = client.completions.create( model="qwen/qwen2.5-0.5b-instruct", prompt="Write a short story about a space explorer.", temperature=0.7, # Moderate temperature for creative writing max_tokens=150, # Longer response for a story top_p=0.9, # Balanced diversity in word choice stop=["\n\n", "THE END"], # Multiple stop sequences presence_penalty=0.3, # Encourage novel elements frequency_penalty=0.3, # Reduce repetitive phrases n=1, # Generate one completion seed=123, # For reproducible results ) print_highlight(f"Response: {response}") ``` #### Returning Routed Experts (MoE Models) For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. ## Structured Outputs (JSON, Regex, EBNF) For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details. ## Using LoRA Adapters SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax. **Server Setup:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path qwen/qwen2.5-0.5b-instruct \ --enable-lora \ --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b ``` For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora). **API Call:** (Recommended) Use the `model:adapter` syntax to specify which adapter to use: ```python Example theme={null} response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct:adapter_a", # ← base-model:adapter-name messages=[{"role": "user", "content": "Convert to SQL: show all users"}], max_tokens=50, ) ``` **Backward Compatible: Using `extra_body`** The old `extra_body` method is still supported for backward compatibility: ```python Example theme={null} # Backward compatible method response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[{"role": "user", "content": "Convert to SQL: show all users"}], extra_body={"lora_path": "adapter_a"}, # ← old method max_tokens=50, ) ``` **Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence. ```python Example theme={null} terminate_process(server_process) ``` # OpenAI APIs - Embedding Source: https://docs.sglang.io/docs/basic_usage/openai_api_embeddings SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings). This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models) ## Launch A Server Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process embedding_process, port = launch_server_cmd( """ sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \ --is-embedding --log-level warning """ ) wait_for_server(f"http://localhost:{port}") ``` ## Using cURL ```python Example theme={null} import subprocess, json text = "Once upon a time" curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \ -H "Content-Type: application/json" \ -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'""" result = subprocess.check_output(curl_text, shell=True) print(result) text_embedding = json.loads(result)["data"][0]["embedding"] print_highlight(f"Text embedding (first 10): {text_embedding[:10]}") ``` ## Using Python Requests ```python Example theme={null} import requests text = "Once upon a time" response = requests.post( f"http://localhost:{port}/v1/embeddings", json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text}, ) text_embedding = response.json()["data"][0]["embedding"] print_highlight(f"Text embedding (first 10): {text_embedding[:10]}") ``` ## Using OpenAI Python Client ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") # Text embedding example response = client.embeddings.create( model="Alibaba-NLP/gte-Qwen2-1.5B-instruct", input=text, ) embedding = response.data[0].embedding[:10] print_highlight(f"Text embedding (first 10): {embedding}") ``` ## Using Input IDs SGLang also supports `input_ids` as input to get the embedding. ```python Example theme={null} import json import os from transformers import AutoTokenizer os.environ["TOKENIZERS_PARALLELISM"] = "false" tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct") input_ids = tokenizer.encode(text) curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \ -H "Content-Type: application/json" \ -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'""" input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][ 0 ]["embedding"] print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}") ``` ## Compact Base64 Responses Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients. ```python Example theme={null} response = requests.post( f"http://localhost:{port}/v1/embeddings", json={ "model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text, "encoding_format": "base64", }, ) base64_embedding = response.json()["data"][0]["embedding"] print_highlight(f"Base64 embedding: {base64_embedding[:20]}...") ``` ```python Example theme={null} terminate_process(embedding_process) ``` ## Multi-Modal Embedding Model Please refer to [Multi-Modal Embedding Model](../supported-models) # OpenAI APIs - Vision Source: https://docs.sglang.io/docs/basic_usage/openai_api_vision SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models. A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision). This tutorial covers the vision APIs for vision language models. SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models/multimodal_language_models). As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py). ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, print_highlight, terminate_process example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" logo_image_url = ( "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png" ) vision_process, port = launch_server_cmd(""" python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning """) wait_for_server(f"http://localhost:{port}", process=vision_process) ``` ## Using cURL Once the server is up, you can send test requests using curl or requests. ```python Example theme={null} import subprocess curl_command = f""" curl -s http://localhost:{port}/v1/chat/completions \\ -H "Content-Type: application/json" \\ -d '{{ "model": "Qwen/Qwen2.5-VL-7B-Instruct", "messages": [ {{ "role": "user", "content": [ {{ "type": "text", "text": "What’s in this image?" }}, {{ "type": "image_url", "image_url": {{ "url": "{example_image_url}" }} }} ] }} ], "max_tokens": 300 }}' """ response = subprocess.check_output(curl_command, shell=True).decode() print_highlight(response) response = subprocess.check_output(curl_command, shell=True).decode() print_highlight(response) ``` ## Using Python Requests ```python Example theme={null} import requests url = f"http://localhost:{port}/v1/chat/completions" data = { "model": "Qwen/Qwen2.5-VL-7B-Instruct", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": {"url": example_image_url}, }, ], } ], "max_tokens": 300, } response = requests.post(url, json=data) print_highlight(response.text) ``` ## Using OpenAI Python Client ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=[ { "role": "user", "content": [ { "type": "text", "text": "What is in this image?", }, { "type": "image_url", "image_url": {"url": example_image_url}, }, ], } ], max_tokens=300, ) print_highlight(response.choices[0].message.content) ``` ## Multiple-Image Inputs The server also supports multiple images and interleaved text and images if the model supports it. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None") response = client.chat.completions.create( model="Qwen/Qwen2.5-VL-7B-Instruct", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": example_image_url, }, }, { "type": "image_url", "image_url": { "url": logo_image_url, }, }, { "type": "text", "text": "I have two very different images. They are not related at all. " "Please describe the first image in one sentence, and then describe the second image in another sentence.", }, ], } ], temperature=0, ) print_highlight(response.choices[0].message.content) ``` ```python Example theme={null} terminate_process(vision_process) ``` # Basic Usage Source: https://docs.sglang.io/docs/basic_usage/overview Core APIs and common usage patterns for SGLang. * [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings * [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration * [Ollama API](./ollama_api) * [Offline Engine API](./offline_engine_api) * [Native API](./native_api) * [Sampling Parameters](./sampling_params) * [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more # Sampling Parameters Source: https://docs.sglang.io/docs/basic_usage/sampling_params This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime. If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions). ## `/generate` Endpoint The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.
Argument Type/Default Description
text `Optional[Union[List[str], str]] = None` The input prompt. Can be a single prompt or a batch of prompts.
input\_ids `Optional[Union[List[List[int]], List[int]]] = None` The token IDs for text; one can specify either text or input\_ids.
input\_embeds `Optional[Union[List[List[List[float]]], List[List[float]]]] = None` The embeddings for input\_ids; one can specify either text, input\_ids, or input\_embeds.
image\_data `Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None` The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal) for details.
audio\_data `Optional[Union[List[AudioDataItem], AudioDataItem]] = None` The audio input. Can be a file name, URL, or base64 encoded string.
sampling\_params `Optional[Union[List[Dict], Dict]] = None` The sampling parameters as described in the sections below.
rid `Optional[Union[List[str], str]] = None` The request ID.
return\_logprob `Optional[Union[List[bool], bool]] = None` Whether to return log probabilities for tokens.
logprob\_start\_len `Optional[Union[List[int], int]] = None` If return\_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.
top\_logprobs\_num `Optional[Union[List[int], int]] = None` If return\_logprob, the number of top logprobs to return at each position.
token\_ids\_logprob `Optional[Union[List[List[int]], List[int]]] = None` If return\_logprob, the token IDs to return logprob for.
return\_text\_in\_logprobs `bool = False` Whether to detokenize tokens in text in the returned logprobs.
stream `bool = False` Whether to stream output.
lora\_path `Optional[Union[List[Optional[str]], Optional[str]]] = None` The path to the LoRA.
custom\_logit\_processor `Optional[Union[List[Optional[str]], str]] = None` Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.
return\_hidden\_states `Union[List[bool], bool] = False` Whether to return hidden states.
return\_routed\_experts `bool = False` Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. With the default `routed_experts_start_len=0`, returns the full available sequence `[0, seqlen - 1)` because RL workflows need routed experts for the full sequence. The result is base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.
routed\_experts\_start\_len `int = 0` If `return_routed_experts`, the absolute start position for returned routed-experts rows. `0` preserves the default full sequence; set it to an accumulated prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. Must be in `[0, prompt_tokens]`.
## Sampling parameters The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs. ### Note on defaults By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`. ```bash Command theme={null} # Use model-provided defaults from generation_config.json (default behavior) python -m sglang.launch_server --model-path --sampling-defaults model # Use SGLang/OpenAI constant defaults instead python -m sglang.launch_server --model-path --sampling-defaults openai ``` ### Core parameters
Argument Type/Default Description
max\_new\_tokens `int = 128` The maximum output length measured in tokens.
stop `Optional[Union[str, List[str]]] = None` One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.
stop\_token\_ids `Optional[List[int]] = None` Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.
stop\_regex `Optional[Union[str, List[str]]] = None` Stop when hitting any of the regex patterns in this list
temperature `float (model default; fallback 1.0)` [Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.
top\_p `float (model default; fallback 1.0)` [Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.
top\_k `int (model default; fallback -1)` [Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.
min\_p `float (model default; fallback 0.0)` [Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.
### Penalizers
Argument Type/Default Description
frequency\_penalty `float = 0.0` Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.
presence\_penalty `float = 0.0` Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.
repetition\_penalty `float = 1.0` Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values \< 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.
min\_new\_tokens `int = 0` Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.
### Constrained decoding Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.
Argument Type/Default Description
json\_schema `Optional[str] = None` JSON schema for structured outputs.
regex `Optional[str] = None` Regex for structured outputs.
ebnf `Optional[str] = None` EBNF for structured outputs.
structural\_tag `Optional[str] = None` The structal tag for structured outputs.
### Other options
Argument Type/Default Description
n `int = 1` Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)
ignore\_eos `bool = False` Don't stop generation when EOS token is sampled.
skip\_special\_tokens `bool = True` Remove special tokens during decoding.
spaces\_between\_special\_tokens `bool = True` Whether or not to add spaces between special tokens during detokenization.
no\_stop\_trim `bool = False` Don't trim stop words or EOS token from the generated text.
custom\_params `Optional[List[Optional[Dict[str, Any]]]] = None` Used when employing `CustomLogitProcessor`. For usage, see below.
## Examples ### Normal Launch a server: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` Detailed example in [send request](./send_request). ### Streaming Send a request and stream the output: ```python Example theme={null} import requests, json response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"].strip() print(output[prev:], end="", flush=True) prev = len(output) print("") ``` Detailed example in [openai compatible api](./openai_api_completions). ### Multimodal Launch a server: ```bash Command theme={null} python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov ``` Download an image: ```bash Command theme={null} curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" "<|im_start|>user\n\nDescribe this image in a very short sentence.<|im_end|>\n" "<|im_start|>assistant\n", "image_data": "example_image.png", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`. Streaming is supported in a similar manner as [above](#streaming). Detailed example in [OpenAI API Vision](./openai_api_vision). ### Structured Outputs (JSON, Regex, EBNF) You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request. SGLang supports two grammar backends: * [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints. * XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md). * [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints. If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar) ``` ```python Example theme={null} import json import requests json_schema = json.dumps({ "type": "object", "properties": { "name": {"type": "string", "pattern": "^[\\w]+$"}, "population": {"type": "integer"}, }, "required": ["name", "population"], }) # JSON (works with both Outlines and XGrammar) response = requests.post( "http://localhost:30000/generate", json={ "text": "Here is the information of the capital of France in the JSON format.\n", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "json_schema": json_schema, }, }, ) print(response.json()) # Regular expression (Outlines backend only) response = requests.post( "http://localhost:30000/generate", json={ "text": "Paris is the capital of", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "regex": "(France|England)", }, }, ) print(response.json()) # EBNF (XGrammar backend only) response = requests.post( "http://localhost:30000/generate", json={ "text": "Write a greeting.", "sampling_params": { "temperature": 0, "max_new_tokens": 64, "ebnf": 'root ::= "Hello" | "Hi" | "Hey"', }, }, ) print(response.json()) ``` Detailed example in [structured outputs](../advanced_features/structured_outputs). ### Custom logit processor Launch a server with `--enable-custom-logit-processor` flag on. ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3-8B-Instruct \ --port 30000 \ --enable-custom-logit-processor ``` Define a custom logit processor that will always sample a specific token id. ```python Example theme={null} from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor class DeterministicLogitProcessor(CustomLogitProcessor): """A dummy logit processor that changes the logits to always sample the given token id. """ def __call__(self, logits, custom_param_list): # Check that the number of logits matches the number of custom parameters assert logits.shape[0] == len(custom_param_list) key = "token_id" for i, param_dict in enumerate(custom_param_list): # Mask all other tokens logits[i, :] = -float("inf") # Assign highest probability to the specified token logits[i, param_dict[key]] = 0.0 return logits ``` Send a request: ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "custom_logit_processor": DeterministicLogitProcessor().to_str(), "sampling_params": { "temperature": 0.0, "max_new_tokens": 32, "custom_params": {"token_id": 5}, }, }, ) print(response.json()) ``` Send an OpenAI chat completion request: ```python Example theme={null} import openai from sglang.utils import print_highlight client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="meta-llama/Meta-Llama-3-8B-Instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0.0, max_tokens=32, extra_body={ "custom_logit_processor": DeterministicLogitProcessor().to_str(), "custom_params": {"token_id": 5}, }, ) print_highlight(f"Response: {response}") ``` # Tutorial: Sending a request Source: https://docs.sglang.io/docs/basic_usage/send_request This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one. * For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision). * For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model). * For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model). ## Launch A Server ```python Example theme={null} from sglang.test.doc_patch import launch_server_cmd from sglang.utils import wait_for_server, terminate_process # This is equivalent to running the following command in your terminal # python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 server_process, port = launch_server_cmd( """ python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \ --host 0.0.0.0 --log-level warning """ ) wait_for_server(f"http://localhost:{port}") ``` ## Using cURL ```python Example theme={null} import subprocess, json curl_command = f""" curl -s http://localhost:{port}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}' """ response = json.loads(subprocess.check_output(curl_command, shell=True)) print(response) ``` ## Using Python Requests ```python Example theme={null} import requests url = f"http://localhost:{port}/v1/chat/completions" data = { "model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` ## Using OpenAI Python Client ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response) ``` ### Streaming ```python Example theme={null} import openai client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None") # Use stream=True for streaming responses response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, stream=True, ) # Handle the streaming output for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` ## Using Native Generation APIs You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params). ```python Example theme={null} import requests response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` ### Streaming ```python Example theme={null} import requests, json response = requests.post( f"http://localhost:{port}/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"] print(output[prev:], end="", flush=True) prev = len(output) ``` ```python Example theme={null} terminate_process(server_process) ``` # Bench Serving Guide Source: https://docs.sglang.io/docs/developer_guide/bench_serving This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs. ### What it does * Generates synthetic or dataset-driven prompts and submits them to a target serving endpoint * Measures throughput, time-to-first-token (TTFT), inter-token latency (ITL), per-request end-to-end latency, and more * Supports streaming or non-streaming modes, rate control, and concurrency limits ### Supported backends and endpoints * `sglang` / `sglang-native`: `POST /generate` * `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions` * `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions` * `sglang-embedding`, `vllm-embedding`: `POST /v1/embeddings` * `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream` * `gserver`: Custom server (Not Implemented yet in this script) * `truss`: `POST /v1/models/model:predict` If `--base-url` is provided, requests are sent to it. Otherwise, `--host` and `--port` are used. When `--model` is not provided, the script will attempt to query `GET /v1/models` for an available model ID (OpenAI-compatible endpoints). ### Prerequisites * Python 3.10+ * Dependencies typically used by this script: `aiohttp`, `numpy`, `requests`, `tqdm`, `transformers`, and for some datasets `datasets`, `pillow`, `pybase64`. Install as needed. * An inference server running and reachable via the endpoints above * If your server requires authentication, set environment variable `OPENAI_API_KEY` (used as `Authorization: Bearer `) ### Quick start Run a basic benchmark against an sglang server exposing `/generate`: ```bash Command theme={null} python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --num-prompts 1000 \ --model meta-llama/Llama-3.1-8B-Instruct ``` Or, using an OpenAI-compatible endpoint (completions): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm \ --base-url http://127.0.0.1:8000 \ --num-prompts 1000 \ --model meta-llama/Llama-3.1-8B-Instruct ``` ### Fair embedding comparison Use the two embedding backends with the same model, tokenizer, input length, prompt count, and concurrency. The benchmark reports input-token throughput and end-to-end latency; embeddings have no decode-side TTFT or TPOT. ```bash Command theme={null} # Start either server on the same hardware and precision, then run one at a time. python3 -m sglang.bench_serving \ --backend sglang-embedding \ --model google/embeddinggemma-300m \ --dataset-name random \ --random-input-len 2048 \ --num-prompts 300 \ --max-concurrency 64 \ --warmup-requests 3 \ --flush-cache ``` ```bash Command theme={null} # vLLM's cache reset endpoint requires VLLM_SERVER_DEV_MODE=1 at server startup. python3 -m sglang.bench_serving \ --backend vllm-embedding \ --model google/embeddinggemma-300m \ --dataset-name random \ --random-input-len 2048 \ --num-prompts 300 \ --max-concurrency 64 \ --warmup-requests 3 \ --flush-cache ``` `--flush-cache` calls `/flush_cache` for SGLang and `/reset_prefix_cache` for vLLM after warmup. For vLLM, start the server with `VLLM_SERVER_DEV_MODE=1`; without it the benchmark fails loudly rather than accidentally measuring warm-cache performance. ### Datasets Select with `--dataset-name`: * `sharegpt` (default): loads ShareGPT-style pairs; optionally restrict with `--sharegpt-context-len` and override outputs with `--sharegpt-output-len` * `random`: random text lengths; sampled from ShareGPT token space * `random-ids`: random token ids (can lead to gibberish) * `image`: generates images and wraps them in chat messages; supports custom resolutions, multiple formats, and different content types * `generated-shared-prefix`: synthetic dataset with shared long system prompts and short questions * `mmmu`: samples from MMMU (Math split) and includes images * `speed-bench`: [SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench) (**SPEculative Evaluation Dataset**) — a unified benchmark for evaluating [Speculative Decoding (SD)](https://arxiv.org/abs/2604.09557) algorithms. Uses the Throughput split, which provides fixed-length input sequences (1K–32K tokens) grouped into three output-entropy categories (`low_entropy`, `mixed`, `high_entropy`). Requires a pre-downloaded JSONL file passed via `--dataset-path`. * `agentic-trace`: replays pre-built multi-turn agentic traces (e.g. OpenHands / SWE-smith). Each conversation is replayed round by round, feeding the server's real assistant reply back into the next round's history. Requires a chat backend (`--backend sglang-oai-chat`) and a trace JSON passed via `--dataset-path`. Common dataset flags: * `--num-prompts N`: number of requests * `--random-input-len`, `--random-output-len`, `--random-range-ratio`: for random/random-ids/image * `--image-count`: Number of images per request (for `image` dataset). * `--apply-chat-template`: apply tokenizer chat template when constructing prompts * `--dataset-path PATH`: file path for ShareGPT json; if blank and missing, it will be downloaded and cached Generated Shared Prefix flags (for `generated-shared-prefix`): * `--gsp-num-groups` * `--gsp-prompts-per-group` * `--gsp-system-prompt-len` * `--gsp-question-len` * `--gsp-output-len` * `--gsp-group-distribution {uniform,zipf}`: per-request prefix-group sampling distribution (default: `uniform`). With `zipf`, each request's group is sampled by rank with `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)`; rank starts at 1 and group index 0 is the hottest. The on-disk dataset cache uses a distinct key per `(group_distribution, zipf_alpha)`, so uniform-mode caches are never mixed with zipf-mode caches. * `--gsp-zipf-alpha FLOAT`: Zipf exponent for `--gsp-group-distribution=zipf`. Must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. Required when the distribution is `zipf`; must be omitted otherwise. Image dataset flags (for `image`): * `--image-count`: Number of images per request * `--image-resolution`: Image resolution; supports presets (4k, 1080p, 720p, 360p) or custom 'heightxwidth' format (e.g., 1080x1920, 512x768) * `--image-format`: Image format (jpeg or png) * `--image-content`: Image content type (random or blank) Agentic trace flags (for `agentic-trace`): * `--dataset-path`: path to the pre-built trace JSON * `--sharegpt-output-len`: per-turn output length (default: 220) * `--dataset-offset`: rotate the conversation list by this many entries before sampling, so successive sweep steps start on fresh conversations * `--agentic-max-turns`: cap each conversation to at most this many turns (useful for small, fast profiling runs) SPEED-Bench flags (for `speed-bench`): * `--dataset-path PATH`: path to the pre-downloaded SPEED-Bench Throughput JSONL (e.g., `throughput_1k.jsonl`). Use the [SPEED-Bench measurement framework](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/specdec_bench) to generate it. * `--speed-bench-category`: filter to one entropy category: `low_entropy`, `mixed`, or `high_entropy` (default: all) * `--speed-bench-output-len`: fixed number of output tokens per request (default: 512) ### Examples 1. To benchmark image dataset with 3 images per request, 500 prompts, 512 input length, and 512 output length, you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-3B-Instruct --disable-radix-cache ``` ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang-oai-chat \ --dataset-name image \ --num-prompts 500 \ --image-count 3 \ --image-resolution 720p \ --random-input-len 512 \ --random-output-len 512 ``` 2. To benchmark random dataset with 3000 prompts, 1024 input length, and 1024 output length, you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path Qwen/Qwen2.5-3B-Instruct ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 3000 \ --random-input 1024 \ --random-output 1024 \ --random-range-ratio 0.5 ``` 3. To benchmark speculative decoding throughput using SPEED-Bench (mixed-entropy category, 1K ISL), you can run: ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \ --speculative-algorithm EAGLE --speculative-draft-model-path ``` ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name speed-bench \ --dataset-path /path/to/throughput_1k.jsonl \ --speed-bench-category mixed \ --speed-bench-output-len 512 \ --num-prompts 512 ``` ### Choosing model and tokenizer * `--model` is required unless the backend exposes `GET /v1/models`, in which case the first model ID is auto-selected. * `--tokenizer` defaults to `--model`. Both can be HF model IDs or local paths. * For ModelScope workflows, setting `SGLANG_USE_MODELSCOPE=true` enables fetching via ModelScope (weights are skipped for speed). * If your tokenizer lacks a chat template, the script warns because token counting can be less robust for gibberish outputs. ### Rate, concurrency, and streaming * `--request-rate`: requests per second. `inf` sends all immediately (burst). Non-infinite rate uses a Poisson process for arrival times. * `--max-concurrency`: caps concurrent in-flight requests regardless of arrival rate. * `--disable-stream`: switch to non-streaming mode when supported; TTFT then equals total latency for chat completions. ### Other key options * `--output-file FILE.jsonl`: append JSONL results to file; auto-named if unspecified * `--output-details`: include per-request arrays (generated texts, errors, ttfts, itls, input/output lens) * `--extra-request-body '{"top_p":0.9,"temperature":0.6}'`: merged into payload (sampling params, etc.) * `--disable-ignore-eos`: pass through EOS behavior (varies by backend) * `--warmup-requests N`: run warmup requests with short output first (default 1) * `--flush-cache`: call `/flush_cache` (sglang) before main run * `--profile`: call `/start_profile` and `/stop_profile` (requires server to enable profiling, e.g., `SGLANG_TORCH_PROFILER_DIR`) * `--lora-name name1 name2 ...`: randomly pick one per request and pass to backend (e.g., `lora_path` for sglang) * `--tokenize-prompt`: send integer IDs instead of text (currently supports `--backend sglang` only) ### Authentication If your target endpoint requires OpenAI-style auth, set: ```bash Command theme={null} export OPENAI_API_KEY=sk-...yourkey... ``` The script will add `Authorization: Bearer $OPENAI_API_KEY` automatically for OpenAI-compatible routes. ### Metrics explained Printed after each run: * Request throughput (req/s) * Input token throughput (tok/s) - includes both text and vision tokens * Output token throughput (tok/s) * Total token throughput (tok/s) - includes both text and vision tokens * Total input text tokens and Total input vision tokens - per-modality breakdown * Concurrency: aggregate time of all requests divided by wall time * End-to-End Latency (ms): mean/median/std/p99 per-request total latency * Time to First Token (TTFT, ms): mean/median/std/p99 for streaming mode * Inter-Token Latency (ITL, ms): mean/median/std/p95/p99/max between tokens * TPOT (ms): Token processing time after first token, i.e., `(latency - ttft)/(tokens-1)` * Accept length (sglang-only, if available): speculative decoding accept length The script also retokenizes generated text with the configured tokenizer and reports "retokenized" counts. ### JSONL output format When `--output-file` is set, one JSON object is appended per run. Base fields: * Arguments summary: backend, dataset, request\_rate, max\_concurrency, etc. * Duration and totals: completed, total\_input\_tokens, total\_output\_tokens, retokenized totals * Throughputs and latency statistics as printed in the console * `accept_length` when available (sglang) With `--output-details`, an extended object also includes arrays: * `input_lens`, `output_lens` * `ttfts`, `itls` (per request: ITL arrays) * `generated_texts`, `errors` ### End-to-end examples 1. sglang native `/generate` (streaming): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.5 \ --num-prompts 2000 \ --request-rate 100 \ --max-concurrency 512 \ --output-file sglang_random.jsonl --output-details ``` 2. OpenAI-compatible Completions (e.g., vLLM): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm \ --base-url http://127.0.0.1:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name sharegpt \ --num-prompts 1000 \ --sharegpt-output-len 256 ``` 3. OpenAI-compatible Chat Completions (streaming): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend vllm-chat \ --base-url http://127.0.0.1:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --num-prompts 500 \ --apply-chat-template ``` 4. Images (VLM) with chat template: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 2 \ --image-resolution 720p \ --random-input-len 128 --random-output-len 256 \ --num-prompts 200 \ --apply-chat-template ``` 4a) Images with custom resolution: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 1 \ --image-resolution 512x768 \ --random-input-len 64 --random-output-len 128 \ --num-prompts 100 \ --apply-chat-template ``` 4b) 1080p images with PNG format and blank content: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model your-vlm-model \ --dataset-name image \ --image-count 1 \ --image-resolution 1080p \ --image-format png \ --image-content blank \ --random-input-len 64 --random-output-len 128 \ --num-prompts 100 \ --apply-chat-template ``` 5. Generated shared prefix (long system prompts + short questions): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name generated-shared-prefix \ --gsp-num-groups 64 --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \ --num-prompts 1024 ``` Zipfian / power-law prefix popularity (opt-in via `--gsp-group-distribution=zipf`): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name generated-shared-prefix \ --gsp-num-groups 64 --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \ --gsp-group-distribution zipf --gsp-zipf-alpha 1.2 \ --seed 42 ``` `zipf` mode samples each request's prefix group from the rank-based distribution `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)` with rank starting at 1, so group index 0 is the hottest. The total request count stays `num_groups * prompts_per_group` — identical to `uniform` mode — and only the per-request group assignment changes. `alpha` must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. The on-disk dataset cache at `~/.cache/sglang/benchmark/gen_shared_prefix_*.pkl` includes `group_distribution` and `zipf_alpha` in its key, so uniform-mode and zipf-mode runs (or two zipf runs with different alpha) never share a cache file. Uniform-mode filenames are unchanged from the legacy format, so existing caches remain valid. This flag controls prefix-popularity shape only. It does not by itself reproduce any production trace or guarantee an observed cache-hit rate for a given engine. 6. Tokenized prompts (ids) for strict length control (sglang only): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --tokenize-prompt \ --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.2 ``` 7. Profiling and cache flush (sglang): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --profile \ --flush-cache ``` 8. TensorRT-LLM streaming endpoint: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend trt \ --base-url http://127.0.0.1:8000 \ --model your-trt-llm-model \ --dataset-name random \ --num-prompts 100 \ --disable-ignore-eos ``` 9. Evaluating large-scale KVCache sharing with mooncake trace (sglang only): ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30000 \ --model model-name \ --dataset-name mooncake \ --mooncake-slowdown-factor 1.0 \ --mooncake-num-rounds 1000 \ --mooncake-workload conversation|mooncake|agent|synthetic --use-trace-timestamps true \ --random-output-len 256 ``` 10. Fake decode stress testing (PD disaggregation, decode-only): When benchmarking pure decode performance in a PD disaggregation setup, you can bypass the prefill node entirely by using `--fake-prefill`. This requires the decode server to be started with `--disaggregation-transfer-backend fake`: ```bash Command theme={null} # Step 1: Start a decode-only server with fake transfer backend python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend fake \ --port 30001 # Step 2: Run bench_serving with --fake-prefill python3 -m sglang.bench_serving \ --backend sglang \ --host 127.0.0.1 --port 30001 \ --model meta-llama/Llama-3.1-8B-Instruct \ --dataset-name random \ --num-prompts 500 \ --random-input-len 1024 --random-output-len 256 \ --fake-prefill ``` Similarly, `bench_one_batch_server` also supports `--fake-prefill`: ```bash Command theme={null} python3 -m sglang.bench_one_batch_server \ --base-url http://127.0.0.1:30001 \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --batch-size 32 --input-len 1024 --output-len 256 \ --fake-prefill ``` The `--fake-prefill` flag automatically injects special sentinel values into each request, telling the decode server to skip real KV transfer and generate fake KV data locally. ### Troubleshooting * All requests failed: verify `--backend`, server URL/port, `--model`, and authentication. Check warmup errors printed by the script. * Throughput seems too low: adjust `--request-rate` and `--max-concurrency`; verify server batch size/scheduling; ensure streaming is enabled if appropriate. * Token counts look odd: prefer chat/instruct models with proper chat templates; otherwise tokenization of gibberish may be inconsistent. * Image/MMMU datasets: ensure you installed extra deps (`pillow`, `datasets`, `pybase64`). * Authentication errors (401/403): set `OPENAI_API_KEY` or disable auth on your server. ### Notes * The script raises the file descriptor soft limit (`RLIMIT_NOFILE`) to help with many concurrent connections. * For sglang, `/server_info` is queried post-run to report speculative decoding accept length when available. # Benchmark and Profiling Source: https://docs.sglang.io/docs/developer_guide/benchmark_and_profiling ## Benchmark SGLang provides four benchmark tools that operate at different levels of the stack. The table below summarizes their key differences:
Tool HTTP Server Scheduler Use Case
bench\_serving Yes (async HTTP client to a running server) Yes (indirectly, via server) Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL)
bench\_one\_batch\_server Yes (sends HTTP requests to a running server) Yes (indirectly, via server) End-to-end single-batch latency including HTTP and scheduler overhead
bench\_offline\_throughput No Yes (directly uses Engine in-process) Maximum throughput measurement without HTTP overhead
bench\_one\_batch No No (directly calls ModelRunner) Kernel-level latency profiling of a single static batch
Use `bench_serving` by default unless there are specific needs. **`bench_serving`** is an async HTTP load-testing client that sends requests at controlled rates with configurable concurrency to a running server. It measures realistic online serving metrics including time-to-first-token (TTFT), time-per-output-token (TPOT), inter-token latency (ITL), and throughput. Use `num-prompts >= 5 * max-concurrency` to measure steady-state performance. Launch a server with `sglang.launch_server` first. ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang --max-concurrency 16 --num-prompts 80 --random-input-len 256 --random-output-len 32 --dataset-name random ``` **`bench_one_batch_server`** sends a single batch as one HTTP request to a running server. Due to only having a single batch, the server is never in a steady-state and metrics will be biased. Launch a server with `sglang.launch_server` first. ```bash Command theme={null} python3 -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 ``` * Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational. * Pass `--lora-name ` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths =`. **`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead. ```bash Command theme={null} python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 ``` **`bench_one_batch`** is the lowest-level tool. It directly instantiates a `ModelRunner` and calls `extend()` / `decode()` on a fixed static batch, bypassing the scheduler entirely. The prefill and decode phases are run separately, making profiling easier but rendering the metrics unrealistic. Because there is no dynamic batching, it may run out of memory for batch sizes that a real server can handle (a real server chunks prefill into smaller batches). This is best suited for profiling individual kernel performance. ```bash Command theme={null} python3 -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 ``` ## Profile with PyTorch Profiler [Pytorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) is a convenient basic tool to inspect kernel execution time, call stack, and kernel overlap and occupancy. ### Profile a server with `sglang.bench_serving` ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # start server python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct # send profiling request from client python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile ``` For `bench_serving --profile`, the output directory is selected on the client side from `--profile-output-dir` or `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`), then sent in the `/start_profile` request. If you call `/start_profile` directly and do not provide `output_dir`, the server uses its own `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`). Setting `SGLANG_TORCH_PROFILER_DIR` on both server and client is still recommended to avoid confusion about where traces are written. For more details, please refer to [Bench Serving Guide](./bench_serving). ### Profile In PD Disaggregation Mode When profiling in PD disaggregation mode, prefill and decode workers **must be profiled separately** due to torch profiler limitations. The `bench_serving` command provides dedicated options for this: #### Profile Prefill Workers ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # start prefill and decode servers (see PD disaggregation docs for setup) python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode prefill python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode decode --port 30001 --base-gpu-id 1 # start router python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000 # send profiling request targeting prefill workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 ``` #### Profile Decode Workers ```bash Command theme={null} # send profiling request targeting decode workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 ``` #### Important Notes * `--profile-prefill-url` and `--profile-decode-url` are **mutually exclusive** - you cannot profile both at the same time * Both options support multiple worker URLs for multi-instance setups: ```bash Command theme={null} # Profile multiple prefill workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 http://127.0.0.1:30002 # Profile multiple decode workers python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 http://127.0.0.1:30003 ``` * Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers * For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation) ### Profile a server with `sglang.bench_offline_throughput` ```bash Command theme={null} export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # profile one batch with bench_one_batch.py # batch size can be controlled with --batch argument python3 -m sglang.bench_one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile # profile multiple batches with bench_offline_throughput.py python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### Profile a server with `sglang.profiler` When the server is running (e.g., processing a decoding request), you can start live profiling immediately by sending a profile request to the server. You can do this by running `python3 -m sglang.profiler`. For example: ```text Output theme={null} # Terminal 1: Send a generation request python3 -m sglang.test.send_one # Terminal 2: Before the above request finishes, quickly launch the following command in a separate terminal. # It will generate a profile of the above request for several decoding batches. python3 -m sglang.profiler ``` You can also combine the above operations into a single command ```text Output theme={null} python3 -m sglang.test.send_one --profile ``` ### Profile a server with HTTP API endpoints SGLang provides HTTP API endpoints to control profiling on a running server. This allows you to start and stop profiling programmatically, which is useful for capturing specific workload patterns. #### Using `/start_profile` endpoint The `/start_profile` endpoint starts profiling on the server. You can control when profiling begins and how long it runs using the following parameters: **Basic usage:** ```bash Command theme={null} # Start profiling immediately for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "num_steps": 10 }' ``` **Parameters:** * `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default * `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/stop_profile` * `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations * `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]` * `merge_profiles` (optional): Whether to merge distributed traces. Default is `false` * `detailed_annotations` (optional): Whether to fold per-iteration request and KV-length aggregates into the trace's `step[...]` markers, for detailed analysis. Default is `false`. See [Detailed annotations](#detailed-annotations) below. **Note on step ranges:** Profiling starts at `start_step` (inclusive) and continues for `num_steps` iterations. For example, with `start_step=3` and `num_steps=10`, profiling captures steps 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 (10 steps total, starting from step 3). **Advanced usage with `start_step`:** ```bash Command theme={null} # Wait 5 steps (warmup), then profile for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", "start_step": 5, "num_steps": 10, "activities": ["CPU", "GPU"] }' ``` **Continuous profiling (manual stop):** ```bash Command theme={null} # Start profiling without num_steps - must manually stop with /stop_profile curl -X POST http://127.0.0.1:30000/start_profile ``` #### Detailed annotations Set `detailed_annotations` to `true` to fold per-iteration aggregates into SGLang's existing per-forward `step[...]` span. For every execution step that runs while profiling is active, SGLang augments that step's marker on the GPU stream with the request and KV-length distribution of the step, so you can reconstruct compute and memory bounds directly from the trace without per-request details. All four per-request aggregates are appended, prefixed by phase — `c_` for context (prefill) and `g_` for generation (decode). The per-phase `sq` is always emitted so each `step[...]` label is self-contained for roofline analysis, even where it duplicates the base label's `bs` (decode) or `toks` (prefill): * `sq`: total query tokens (`Σ N_Q`) * `sqsq`: sum of squared query tokens per request (`Σ N_Q²`) * `sqsk`: sum of query·KV tokens per request (`Σ N_Q·N_KV`) * `sk`: total KV tokens (`Σ N_KV`) A pure prefill (`EXTEND`) or decode (`DECODE`) forward emits a single group; a mixed forward emits both, with `c=`/`g=` request counts. Example labels: ```text theme={null} step[EXTEND bs=1 toks=1025 c_sq=1025 c_sqsq=1050625 c_sqsk=1050625 c_sk=1025] step[DECODE bs=64 g_sq=64 g_sqsq=64 g_sqsk=100032 g_sk=100032] step[MIXED bs=66 c=2 g=64 c_sq=2048 c_sk=2048 c_sqsq=2097152 c_sqsk=2097152 g_sq=64 g_sk=65600 g_sqsq=64 g_sqsk=65600] ``` With speculative decoding (EAGLE/MTP) each request contributes multiple query tokens per step, so `sq` no longer equals `bs`. Both draft-decode and target-verify (`TARGET_VERIFY`) steps are emitted in the **generation** group. For example, a verify step with 3 draft tokens across 2 requests (`seq_lens=[10, 20]`): ```text theme={null} step[TARGET_VERIFY bs=2 g_sq=6 g_sqsq=18 g_sqsk=90 g_sk=30] ``` ```bash Command theme={null} # Profile 10 steps with detailed annotations enabled curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", "num_steps": 10, "activities": ["CPU", "GPU"], "detailed_annotations": true }' ``` The annotations only appear when profiling is active with `detailed_annotations` enabled, so they add no overhead on the normal serving path. The behavior is identical in eager and CUDA graph modes. When viewing the trace (see [View traces](#view-traces)), the augmented `step[...]` markers appear on the GPU stream alongside the kernels for each step. #### Using `/stop_profile` endpoint The `/stop_profile` endpoint stops an ongoing profiling session and saves the trace file. ```bash Command theme={null} # Stop profiling and save traces curl -X POST http://127.0.0.1:30000/stop_profile ``` This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps. #### Example workflow ```bash Command theme={null} # Terminal 1: Start the server export SGLANG_TORCH_PROFILER_DIR=/tmp/profiles python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct # Terminal 2: Start continuous profiling curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "start_step": 3 }' # Terminal 3: Send requests to generate load python -m sglang.bench_serving --backend sglang --num-prompts 100 # Terminal 2: Stop profiling when done curl -X POST http://127.0.0.1:30000/stop_profile ``` ### Profiler Trace Merger for Distributed Traces SGLang now supports automatic merging of profiling traces from distributed setups with multiple parallelism types (TP, DP, PP, EP). This feature is particularly useful for analyzing performance across distributed runs. #### Multi-Node Profiling and Shared Storage Considerations Single-node profiler output merging is completely supported. When profiling in distributed environments spanning multiple nodes, shared storage (e.g., NFS, Lustre) should be accessible by all nodes for the output directory to enable merging of trace files. If there is no shared storage accessible across nodes, automatic merging of trace files during profiling is not supported directly as of now. #### HTTP API Usage ```bash Command theme={null} # Start profiling with automatic trace merging enabled curl -X POST /start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "/tmp/profiles", # where to store profile traces "num_steps": 10, "activities": ["CPU", "GPU"], "merge_profiles": true # optional argument to merge profile traces (default=False) }' ``` #### Command Line Usage ```bash Command theme={null} # Start profiling with merge enabled python -m sglang.profiler \ --num-steps 10 \ --cpu \ --gpu \ --output-dir /tmp/profiles \ --merge-profiles # optional argument to merge profile traces (default=False) ``` #### Output Files The profile merger generates: * Individual rank trace files: `{profile_id}-TP-{tp}-DP-{dp}-PP-{pp}-EP-{ep}.trace.json.gz` * Merged trace file: `merged-{profile_id}.trace.json.gz` ### Profile the CUDA graph capture phase The tools above profile the steady-state runtime (prefill / decode). To instead profile the **CUDA graph capture phase** that runs once at server startup, launch the server with `--enable-profile-cuda-graph`. This runs a PyTorch Profiler pass over the decode CUDA-graph capture, which is useful for diagnosing slow or memory-heavy graph capture. `--enable-profile-cuda-graph` (server arg) builds the capture profiler and always emits the per-kernel CPU/CUDA time summary tables and a CUDA memory snapshot. Persisting Chrome traces to disk is opt-in via one of two env vars (both no-ops unless `--enable-profile-cuda-graph` is also set): * `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1` — writes **one combined trace per tensor-parallel rank** for the whole capture pass, named `cuda_graph_capture-{runner}-TP-{tp_rank}.json.gz`. * `SGLANG_GRAPH_BATCH_CAPTURE=1` — writes **one trace per captured batch size per rank**, named `{runner}_bs_{bs}_rank{tp_rank}.json.gz`. The profiler runs on a `wait=2, warmup=0, active=1` schedule (the two dummy runs before each capture are skipped) with `record_shapes`, `with_stack`, `with_flops`, and `profile_memory` enabled, giving per-shape kernel identities, input shapes, FLOPs, and memory. If both env vars are set, `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE` (the single combined trace) takes precedence. ```bash Command theme={null} # set trace path export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # opt in to per-batch-size capture traces (or set # SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1 for a single combined trace per rank) export SGLANG_GRAPH_BATCH_CAPTURE=1 # launch the server with CUDA graph capture profiling enabled python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --enable-profile-cuda-graph ``` Behavior and output: * All traces are written to `${SGLANG_TORCH_PROFILER_DIR}/graph_capture_profile/` (defaults to `/tmp/graph_capture_profile/` if the variable is unset). Files are namespaced by runner class and TP rank so concurrent capture passes (e.g. EAGLE target/draft/draft-extend) and ranks don't collide. * A CUDA memory snapshot (`cuda_graph_runner_memory_usage.pickle`) and per-kernel CPU/CUDA time summary tables are always emitted for the capture phase (independent of the env vars above). * Only the decode CUDA-graph runner is profiled. The capture traces are viewed the same way as other PyTorch Profiler traces (see [View traces](#view-traces)). ### Possible PyTorch bugs If in any cases you encounter the following error (for example, using qwen 2.5 VL): ```bash Command theme={null} RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty. ``` This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows: ```bash Command theme={null} export SGLANG_PROFILE_WITH_STACK=False python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### View traces Trace files can be loaded and visualized from: 1. [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (any browser) 2. chrome://tracing (Chrome browser only) If browser cannot open trace file due to its large size, client can generate a small trace file (\<100MB) by controlling number of prompts and lengths of prompt outputs. For example, when profiling a server, ```bash Command theme={null} python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 2 --sharegpt-output-len 100 --profile ``` This command sets the number of prompts to 2 with `--num-prompts` argument and limits the length of output sequences to 100 with `--sharegpt-output-len` argument, which can generate a small trace file for browser to open smoothly. Additionally, if you want to locate the SGLang Python source code through the cuda kernel in Trace, you need to disable CUDA Graph when starting the service. This can be done by using the `--disable-cuda-graph` parameter in the command to start the service. ## Profile with Nsight [Nsight systems](https://docs.nvidia.com/nsight-systems/) is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events. 1. Prerequisite: Install using apt, or run inside a [NVIDIA Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) or [SGLang Docker container](https://github.com/sgl-project/sglang/tree/main/docker). ```bash Command theme={null} # install nsys # https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html apt update apt install -y --no-install-recommends gnupg echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub apt update apt install nsight-systems-cli ``` 2. To profile a single batch, use ```bash Command theme={null} nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512 ``` 3. To profile a server, e.g. ```bash Command theme={null} # launch the server, set the delay and duration times according to needs # after the duration time has been used up, server will be killed by nsys nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o sglang.out --delay 60 --duration 70 python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disable-radix-cache # client python3 -m sglang.bench_serving --backend sglang --num-prompts 1000 --dataset-name random --random-input 1024 --random-output 512 ``` In practice, we recommend users to set `--duration` argument to a large value. Whenever user wants the server to stop profiling. Firstly run: ```bash Command theme={null} nsys sessions list ``` to get the session id in the form of `profile-XXXXX`, then run: ```bash Command theme={null} nsys stop --session=profile-XXXXX ``` to manually kill the profiler and generate `nsys-rep` files instantly. 4. Use NVTX to annotate code regions, e.g. to see their execution time. ```bash Command theme={null} # install nvtx pip install nvtx ``` ```python Example theme={null} # code snippets import nvtx with nvtx.annotate("description", color="color"): # some critical code ``` ### Layer-wise NVTX Profiling with Nsight Systems SGLang provides built-in layerwise NVTX annotations that can be combined with the CUDA Profiler for detailed per-layer profiling in Nsight Systems. This is particularly useful for identifying performance bottlenecks at the layer level. #### Using `--enable-layerwise-nvtx-marker` with Nsight Systems and `/start_profile` The `--enable-layerwise-nvtx-marker` flag automatically adds NVTX markers to every layer in your model. This is particularly powerful when combined with Nsight Systems profiling to see detailed per-layer performance. **Method 1: Using `/start_profile` with CUDA\_PROFILER (for programmatic control)** This method allows you to control exactly when profiling starts/stops via HTTP API while Nsight Systems is running. 1. Launch the server with layerwise NVTX enabled under Nsight Systems: ```bash Command theme={null} # Terminal 1: Start server with nsys and capture-range option nsys profile --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --capture-range=cudaProfilerApi \ --capture-range-end=stop \ -o layerwise_profile \ python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --enable-layerwise-nvtx-marker \ --disable-cuda-graph ``` Note: NVTX markers are not emitted for kernel launches captured by CUDA graphs. Use `--disable-cuda-graph` to ensure all layerwise NVTX markers are emitted in the trace. 2. In another terminal, control profiling via `/start_profile` with `CUDA_PROFILER` activity: ```bash Command theme={null} # Terminal 2: Wait for server to be ready, then start CUDA profiling # Wait 3 steps for warmup, then profile for 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "start_step": 3, "num_steps": 10, "activities": ["CUDA_PROFILER"] }' ``` 3. Send requests to generate load: ```bash Command theme={null} # Terminal 3: Generate workload python -m sglang.bench_serving --backend sglang --num-prompts 100 ``` 4. Profiling will automatically stop after 10 steps (due to `num_steps: 10`). If you hadn't specified `num_steps`, you would need to manually stop it: ```bash Command theme={null} # Terminal 2: Only needed if num_steps was not specified curl -X POST http://127.0.0.1:30000/stop_profile ``` The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/stop_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead. **Method 2: Simpler approach without `/start_profile` API** For simpler use cases where you don't need fine-grained control over profiling start/stop, you can profile with Nsight Systems capturing the entire workload: ```bash Command theme={null} # Terminal 1: Start server with layerwise NVTX # Note: --disable-cuda-graph ensures all NVTX markers are emitted python -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --enable-layerwise-nvtx-marker \ --disable-cuda-graph # Terminal 2: Profile the benchmarking client nsys profile --trace-fork-before-exec=true \ --cuda-graph-trace=node \ -o layerwise_profile \ python -m sglang.bench_serving --backend sglang --num-prompts 10 ``` This approach profiles the entire client execution, including all server interactions. The layerwise NVTX markers will be visible in the Nsight Systems timeline. **Viewing the profiling results:** Open the generated `.qdrep` file with Nsight Systems: ```bash Command theme={null} nsys-ui layerwise_profile.qdrep ``` In the Nsight Systems GUI, you'll see: * **NVTX ranges**: Each layer appears as a labeled range in the timeline with detailed information in the marker metadata * **CUDA kernels**: All GPU kernels are shown alongside the layer annotations * **Layer hierarchy**: The full module path (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct.model.layers.0.self_attn.qkv_proj`) helps identify specific layers. The prefix uses the full model path from `--model-path`. * **Tensor shapes**: Input/output dimensions and parameter shapes are included in the NVTX marker data **Benefits of layerwise NVTX profiling:** * **Granular visibility**: See exactly which layers are taking the most time * **Memory tracking**: Identify layers with large memory allocations * **Bottleneck identification**: Quickly locate inefficient operations * **Communication overhead**: In multi-GPU setups, see per-layer communication costs * **Development debugging**: Validate that model architecture changes have the expected performance impact ## Other tips 1. You can benchmark a model using dummy weights by only providing the config.json file. This allows for quick testing of model variants without training. To do so, add `--load-format dummy` to the above commands and then you only need a correct `config.json` under the checkpoint folder. 2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using: ```bash Command theme={null} python -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}' ``` 3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing) 4. For more arguments see [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html). # Contribution Guide Source: https://docs.sglang.io/docs/developer_guide/contribution_guide Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process. ## Install SGLang from Source ### Fork and clone the repository **Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally. ```bash theme={null} git clone https://github.com//sglang.git ``` ### Build from source Refer to [Install SGLang from Source](../get-started/install#method-2-from-source). ## Format code with pre-commit We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run: ```bash theme={null} pip3 install pre-commit pre-commit install pre-commit run --all-files ``` * **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. * **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. * Documentation links and anchors are checked with Mintlify in CI. To run the same version locally, install it with `npm install -g mint@4.2.559`, then run `cd docs && mint broken-links --check-anchors --check-redirects`. * The manual `lychee` pre-commit hook checks links in the repository-level `README.md`: `pre-commit run --hook-stage manual lychee --all-files`. ## Run and add unit tests If you add a feature or fix a bug, add focused regression coverage when the test protects a concrete behavior, invariant, or bookkeeping contract. ### Unit tests (no server required) Unit tests live under [`test/registered/unit/`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit), organized to mirror the `python/sglang/srt/` source tree. These tests validate component logic **without** launching a server or loading real model weights. SGLang supports both Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework and [pytest](https://docs.pytest.org/). Registered CI tests must use `CustomTestCase` instead of raw `unittest.TestCase`, register themselves with `register_*_ci(...)`, and include a standard `__main__` entry point. CI discovers registrations with `test/run_suite.py` and executes each selected test file directly with fail-fast enabled. **When to add a unit test:** If you modify a file under `python/sglang/srt/`, check whether a corresponding test exists in `test/registered/unit/` and add coverage for your changes. For example: ```text theme={null} srt/mem_cache/radix_cache.py → unit/mem_cache/test_radix_cache_unit.py srt/sampling/sampling_params.py → unit/sampling/test_sampling_params.py ``` **Run tests locally:** ```bash theme={null} # Closest to how CI executes an individual registered test file python3 test/registered/unit/mem_cache/test_radix_cache_unit.py # Run a registered CI suite python3 test/run_suite.py --hw cpu --suite base-a-test-cpu # Pytest remains useful for local directory-level discovery pytest test/registered/unit/mem_cache/ -v ``` **Run with coverage:** ```bash Command theme={null} pytest test/registered/unit/ --cov --cov-config=.coveragerc -v ``` For conventions on CI registration, test structure, and examples, see [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit/README.md). ### E2E tests (server required) For tests that require launching a server, refer to [`test/registered/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/README.md) for guidance on where to place your test. For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). ## Write documentation Documentation is a good way for new contributors to learn the SGLang codebase. See [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md) for the current Mintlify setup and validation commands. ## Test the accuracy If your code changes model output, run an accuracy evaluation appropriate for the affected model and feature. For example, after launching a server, run the unified GSM8K evaluator: ```bash theme={null} # Launch a server sglang serve --model-path Qwen/Qwen3-8B # Evaluate python3 -m sglang.test.run_eval \ --eval-name gsm8k \ --port 30000 \ --num-examples 200 ``` Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine. Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test. GSM8K is too easy for many state-of-the-art models, so use a more challenging or feature-specific evaluation when appropriate. See [Evaluating New Models with SGLang](/docs/developer_guide/evaluating_new_models) for current MMLU, GSM8K, HellaSwag, GPQA, HumanEval, and MMMU commands. Additional examples are available in [test/manual/eval](https://github.com/sgl-project/sglang/tree/main/test/manual/eval). ## Benchmark the speed Refer to [Benchmark and Profiling](./benchmark_and_profiling). ## Requesting a review for merge You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. Then your PR can be merged. ## How to Trigger CI Tests We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) **PR authors** can use `/rerun-failed-ci` on their own PRs even if they are not listed in `CI_PERMISSIONS.json`. Selective reruns have additional rules because they execute PR code on self-hosted runners; see the permission table below. For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: * `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). * `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed or skipped**. * `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). * `/rerun-test [ ...]`: Reruns one or more specific tests directly. A spec may select a file, class, or method using `::[.]`. Multiple specs and file globs are supported. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py`, or `/rerun-test test_*backend*.py`. * `/rerun-group [ ...]`: Expands one or more registered test groups (for example, `/rerun-group hicache`) and dispatches their tests through the same selective-rerun workflow. The rerun commands have the following permission rules: | Command | PR author | Other users | Additional rule for fork PRs | | ----------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | | `/rerun-failed-ci` | Always allowed on their own PR | Must have `can_rerun_failed_ci` in `CI_PERMISSIONS.json` | None | | `/rerun-test`, `/rerun-group` | No special allowance; judged as any other commenter | Must have `cooldown_interval_minutes: 0` in `CI_PERMISSIONS.json`, or `write`/`admin` repository permission | None -- the same rule applies wherever the PR comes from | Selective reruns do not build or install a PR-local `sglang-kernel` wheel. If a PR changes `python/sglang/kernels/aot/` or code that depends on that AOT change, use the normal/full PR CI workflow. A `/rerun-test` or `/rerun-group` result may use the pinned released kernel and does not validate the combined change. If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). If you don’t have permission and you’re not the PR author, please ask maintainers to trigger CI for you. ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: ```yaml Config theme={null} cool-down-minutes: description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting" type: number default: 120 ``` Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. ## Code style guidance * Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. * Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. * Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible. * Make functions as pure as possible. Avoid in-place modification of arguments. * Prefer immutable data and compute configuration-derived values once during initialization when their inputs cannot change. * Keep functions under roughly 100 lines and make orchestration functions read like high-level pseudocode by extracting details into focused helpers. * Keep files concise. If a file exceeds 2,000 lines of code, split it into cohesive smaller modules. * Avoid adding mixins; prefer composition or plain functions. Use `msgspec.Struct` for new data containers instead of `dataclasses.dataclass` or `attrs`. * Prefer keyword arguments for calls with two or more arguments, and pass callees the specific values they need instead of a large state-holding object. * In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. * Keep tests run fast. * If a single test file runs longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). * If a single job in a GitHub Actions workflow runs longer than 30 minutes, split it into smaller jobs or steps. * Reuse server launches across test methods in E2E test files. * Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python's [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. * When supporting new hardware or features, follow these guidelines: * Do not drastically change existing code. * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`). * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. ## How to update kernels in SGLang Choose the implementation and release path based on the kernel's dependencies. For a lightweight kernel that does not depend on CUTLASS or another large C++ project, prefer the in-tree JIT kernel path. Use the AOT `sglang-kernel` path for heavyweight kernels, large C++ dependencies, or operations that need wheel packaging and Torch operator registration. FlashInfer-based kernels are an exception and may still use the JIT path. SGLang also consumes separately released custom builds of DeepGEMM and DeepEP; update those in their source repositories as described below. ### Update sglang-kernel The `sglang-kernel` distribution (formerly `sgl-kernel`) is a separate Python package, but its source now lives in this repository under `python/sglang/kernels/aot/`. Normal PR CI builds and installs a PR-local `sglang-kernel` wheel, so it can test an AOT kernel and its caller together. This does not establish compatibility after merge: installed SGLang and scheduled CI use the released version pinned in `python/pyproject.toml`. If a caller unconditionally requires a new operator or a changed kernel contract, first merge and release the kernel, update the pinned `sglang-kernel` version, and then land the caller. A combined PR is acceptable only when every supported path remains correct with the pinned wheel—for example, when the caller checks operator availability and retains a semantically equivalent fallback until the released wheel includes the change. Selective `/rerun-test` and `/rerun-group` workflows do not install PR-local wheels and cannot validate the combined AOT path. For an AOT kernel change: 1. Implement the kernel under `python/sglang/kernels/aot/csrc/` and update its declaration, Torch registration, and CMake source list. 2. Expose the Python API under `python/sglang/kernels/aot/python/sgl_kernel/`. 3. Add correctness tests under `python/sglang/kernels/aot/tests/` and a benchmark under `python/sglang/kernels/aot/benchmark/` when applicable. 4. Build and test from `python/sglang/kernels/aot/` following its [README](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/aot/README.md). Do not bump the pinned `sglang-kernel` version in `python/pyproject.toml` merely to test a PR-local kernel change. ### Update sgl-deep-gemm Develop SGLang's customized DeepGEMM package on the [`dev` branch of `sgl-project/DeepGEMM`](https://github.com/sgl-project/DeepGEMM/tree/dev). Rebase incoming changes onto `dev`, place new or modified package tests under `sgl_deep_gemm/tests/`, and follow the [`sgl-deep-gemm` README](https://github.com/sgl-project/DeepGEMM/blob/dev/sgl_deep_gemm/README.md) to build and install a local wheel. After the implementation is merged, ask the SGLang team to run the [sgl-deep-gemm release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepgemm.yml) with the new version, CUDA target, and DeepGEMM branch. Once all required wheels are published and verified, update the `sgl-deep-gemm` pin in `python/pyproject.toml` before landing SGLang code that requires the new or changed behavior. ### Update sgl-deep-ep Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use the implementation branch for the target platform: `sgl-deepep` for CUDA 13 on x86\_64 or aarch64, `sgl-deepep-cu12-x86` for CUDA 12.9 on x86\_64, or `sgl-deepep-cu12-arm` for CUDA 12.9 on aarch64. Merge packaging changes into `sgl-deepep-packaging`. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep-packaging/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix. To validate locally, check out the selected implementation branch as `DeepEP-source` and the packaging branch as `DeepEP-packaging`. Install the required build dependencies first; CUDA 12.9 builds also require GDRCopy. The following CUDA 13 example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds: ```bash theme={null} DEEPEP_OUTPUT_DIR="$(mktemp -d)" bash DeepEP-packaging/sgl_deep_ep/build_sgl_deep_ep.sh \ DeepEP-source \ DeepEP-packaging/sgl_deep_ep \ "${DEEPEP_OUTPUT_DIR}" \ 13.0 \ "$(uname -m)" python3 -m pip install --force-reinstall --no-deps \ "${DEEPEP_OUTPUT_DIR}"/sgl_deep_ep-*.whl python3 -c "import deep_ep; print(deep_ep.__file__)" ``` Use `12.9` instead of `13.0` for a CUDA 12.9 build. The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run the test appropriate for the implementation branch: ```bash theme={null} # CUDA 13 implementation branch python3 DeepEP-source/tests/elastic/test_ep.py --num-processes 8 # CUDA 12.9 implementation branches python3 DeepEP-source/tests/test_intranode.py --num-processes 8 ``` Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version, CUDA target, and packaging ref. After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes. ## Tips for newcomers If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out the following materials as startup guide: * [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang. * [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. * [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance. If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). Thank you for your interest in SGLang. Happy coding! # Development Guide Using Docker Source: https://docs.sglang.io/docs/developer_guide/development_guide_using_docker ## Setup VSCode on a Remote Host (Optional - you can skip this step if you plan to run sglang dev container locally) 1. In the remote host, download `code` from [VSCode](https://code.visualstudio.com/download) and run `code tunnel` in a shell. Example ```bash Command theme={null} wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz tar xf vscode_cli_alpine_x64_cli.tar.gz # https://code.visualstudio.com/docs/remote/tunnels ./code tunnel ``` 2. In your local machine, press F1 in VSCode and choose "Remote Tunnels: Connect to Tunnel". ## Setup Docker Container ### Option 1. Use the default dev container automatically from VSCode There is a `.devcontainer` folder in the sglang repository root folder to allow VSCode to automatically start up within dev container. You can read more about this VSCode extension in VSCode official document [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers). VSCode Dev Container Architecture *Figure 1: Diagram from VSCode official documentation [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).* To enable this, you only need to: 1. Start Visual Studio Code and install [VSCode dev container extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers). 2. Press F1, type and choose "Dev Container: Open Folder in Container. 3. Input the `sglang` local repo path in your machine and press enter. The first time you open it in dev container might take longer due to docker pull and build. Once it's successful, you should set on your status bar at the bottom left displaying that you are in a dev container: VSCode Dev Container Status Bar Now when you run `sglang.launch_server` in the VSCode terminal or start debugging using F5, sglang server will be started in the dev container with all your local changes applied automatically: SGLang Server Running in Dev Container ### Option 2. Start up containers manually (advanced) The following startup command is an example for internal development by the SGLang team. You can **modify or add directory mappings as needed**, especially for model weight downloads, to prevent repeated downloads by different Docker containers. ❗️ **Note on RDMA** 1. `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them but keeping them there does not harm. Thus, we enable these two flags by default in the commands below. 2. You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`. ```bash Command theme={null} # Change the name to yours docker run -itd --shm-size 32g --gpus all -v --ipc=host --network=host --privileged --name sglang_dev lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_dev /bin/zsh ``` Some useful volumes to mount are: 1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`. 2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer. Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer. ```bash Command theme={null} docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_zhyncs /bin/zsh ``` Example 2: Mounting both HuggingFace cache and local SGLang repo. Local code changes are automatically synced to the devcontainer as the SGLang is installed in editable mode in the dev image. ```bash Command theme={null} docker run -itd --shm-size 32g --gpus all -v $HOME/.cache/huggingface/:/root/.cache/huggingface -v $HOME/src/sglang:/sgl-workspace/sglang --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh docker exec -it sglang_zhyncs /bin/zsh ``` ## Debug SGLang with VSCode Debugger 1. (Create if not exist) open `launch.json` in VSCode. 2. Add the following config and save. Please note that you can edit the script as needed to apply different parameters or debug a different program (e.g. benchmark script). ```JSON Config theme={null} { "version": "0.2.0", "configurations": [ { "name": "Python Debugger: launch_server", "type": "debugpy", "request": "launch", "module": "sglang.launch_server", "console": "integratedTerminal", "args": [ "--model-path", "meta-llama/Llama-3.2-1B", "--host", "0.0.0.0", "--port", "30000", "--trust-remote-code", ], "justMyCode": false } ] } ``` 3. Press "F5" to start. VSCode debugger will ensure that the program will pause at the breakpoints even if the program is running at remote SSH/Tunnel host + dev container. ## Profile ```bash Command theme={null} # Change batch size, input, output and add `disable-cuda-graph` (for easier analysis) # e.g. DeepSeek V3 nsys profile -o deepseek_v3 python3 -m sglang.bench_one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph ``` ## Evaluation ```bash Command theme={null} # e.g. gsm8k 8 shot python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 2000 --num-shots 8 ``` # Development Guide for JIT Kernels Source: https://docs.sglang.io/docs/developer_guide/development_jit_kernel_guide ## Environment Setup We strongly recommend using `clangd` as the language server for JIT kernel development. For Ubuntu/Debian, you can download clangd from [apt.llvm.org](https://apt.llvm.org/). If you are using VS Code, we recommend installing the `clangd` extension for better IDE integration. All JIT-related files are located in `python/sglang/kernels/jit`. Unlike `sgl-kernel`, which compiles CUDA/C++ binaries ahead of time (AOT), just-in-time (JIT) kernels are compiled at runtime. Consequently, a static `compile_commands.json` cannot be generated. To enable code completion with `clangd`, run `python -m sglang.kernels.jit` to generate a `.clangd` configuration file in your current directory. After generating the file, restart the clangd language server. It should now recognize all JIT kernel files. ## Code Structure ### C++ Implementation C++ source code is located in `python/sglang/kernels/jit/csrc`. Reusable functions should be placed in `python/sglang/kernels/jit/include`. JIT C++ lives in `namespace sglang`: open it after the include block and close it at the end of the file, with the device kernels and the host wrapper both inside. The shared `host::` and `device::` helpers are nested in it as well, so they resolve unqualified and need no `sglang::` prefix. We use [tvm-ffi](https://github.com/apache/tvm-ffi) for efficient foreign language bindings. Refer to the [documentation](https://tvm.apache.org/ffi/) for advanced usage, such as exporting C++ objects. Typically, `tvm::ffi::TensorView` is sufficient for passing PyTorch Tensors from Python. ### Python Interface Python interfaces are defined in `python/sglang/kernels/jit`. The `load_jit` utility function in `python/sglang/kernels/jit/utils/compile.py` loads and returns the compiled module. To export a C++ function (e.g., `cpp_func`), pass `cuda_wrappers=[("func", "cpp_func")]` to `load_jit`. The function can then be called in Python as `module.func`. `load_jit` emits the export wrapper inside `namespace sglang`, so write `cpp_func` without a `sglang::` prefix. For caching compiled modules, prefer `sglang.kernels.jit.utils.cache_once` over `functools.lru_cache`. `functools.lru_cache` is not compatible with `torch.compile`. ### C++ Utilities The following C++ utilities are available: #### Integer Range Similar to PyTorch, we provide an `irange` function to represent an integer range. ```C++ Example theme={null} #include void test() { for (auto i : host::irange(100)) { // [0, 100) // do something } for (auto i : host::irange(0, 100)) { // [0, 100) // do something } } ``` #### Runtime Checking `CHECK_HOST` is the preferred runtime check: stream-style, and zero overhead when the check passes — the message expressions are only evaluated on failure. `RuntimeCheck` is the function-style alternative; note its message arguments are always evaluated, even when the check passes. `RuntimeDeviceCheck` verifies the status of the last kernel launch, and `CHECK_CUDA` is its stream-style equivalent for checking a `cudaError_t` with extra context. ```C++ Example theme={null} #include #include void test() { CHECK_HOST(1 + 1 == 2) << 1 + 1 << " != " << 2; // preferred host::RuntimeCheck(1 + 1 == 2, 1 + 1, " != ", 2); host::RuntimeDeviceCheck(); // check the provided `cudaError_t` host::RuntimeDeviceCheck(cudaGetLastError()); CHECK_CUDA(cudaGetLastError()) << "after my_kernel launch"; } ``` #### Tensor Checking `TensorMatcher` provides a readable way to validate and extract tensor shape information. ```cpp Example theme={null} #include void test(const tvm::ffi::TensorView k_cache, const tvm::ffi::TensorView v_cache) { using namespace host; auto D = SymbolicSize{"D"}; // cache dimension auto N = SymbolicSize{"N"}; // kvcache stride auto dtype = SymbolicDType{}; auto device = SymbolicDevice{}; TensorMatcher({-1, D}) // .with_strides({N, 1}) .with_dtype(dtype) .with_device(device) .verify(k_cache) .verify(v_cache); } ``` Configure the `TensorMatcher` with expected stride, dtype, and device properties before verification. * If `with_strides` is omitted, the tensor is expected to be contiguous. * Template arguments in `with_dtype` restrict the allowed data types. * Template arguments in `with_device` restrict the allowed devices. * Values passed to `with_xxx` methods enforce equality checks. * Passing `-1` for size or stride allows matching any value. A `Symbolic` variable must resolve to the same value across all verifications. Use `.unwrap()` to retrieve the matched value after verification. > Note: `TensorMatcher` is a temporary expression and should not be stored in a variable. > Tip: Add `//` at the end of the `TensorMatcher` chain to enforce proper indentation. #### Kernel Launching `LaunchKernel::resolve_device` retrieves the current `cudaStream` from PyTorch. Kernels can also be launched directly using `LaunchKernel`. ```cpp Example theme={null} #include #include __global__ void kernel() {} void test() { const auto num_blocks = 1; const auto num_threads = 32; const auto dynamic_smem = 0; DLDevice dev; // suppose this is initialized properly host::LaunchKernel(num_blocks, num_threads, dev)(kernel); cudaStream_t stream = host::LaunchKernel::resolve_device(dev); host::LaunchKernel(num_blocks, num_threads, stream, dynamic_smem)(kernel); } ``` ## Add new kernels This section walks through a complete, end-to-end example of adding a new JIT kernel to the system. We use a simple add\_constant kernel as a running example, which adds a constant integer value to every element of an input tensor. Conceptually, the Python interface looks like this: ```python Example theme={null} def add_constant(src: torch.Tensor, c: int): return src + c ``` ### STEP 1: Write the C++ kernel Write your CUDA kernel in [kernels/jit/csrc/add\_constant.cuh](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/jit/csrc/add_constant.cuh). For demonstration purposes, we pass the constant value as a template parameter. ```cpp Example theme={null} #include // For TensorMatcher, SymbolicSize, SymbolicDevice #include // For LaunchKernel #include // For div_ceil, CHECK_HOST #include #include #include #include namespace sglang { template __global__ void add_constant_kernel(int32_t* dst, const int32_t* src, size_t length) { size_t idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx < length) { dst[idx] = src[idx] + kConstant; } } constexpr size_t kBlockSize = 256; // You can also use struct with static method as an alternative template void add_constant(tvm::ffi::TensorView dst, tvm::ffi::TensorView src) { using namespace host; // 1. Validate input tensors SymbolicSize N = {"num_elements"}; SymbolicDevice device_; TensorMatcher({N}) // 1D tensor, must be contiguous .with_dtype() // must be int32 .with_device(device_) // must be on CUDA device .verify(dst) // check tensor dst .verify(src); // check tensor src // 2. Extract required parameters, prepare for kernel launch const size_t num_elements = N.unwrap(); const size_t grid_size = div_ceil(num_elements, kBlockSize); const DLDevice device = device_.unwrap(); // some extra runtime checks using CHECK_HOST CHECK_HOST(num_elements > 0) << "We only support non-empty tensors, got num_elements = " << num_elements; // 3. Launch the kernel. Error code will be automatically checked. LaunchKernel(grid_size, kBlockSize, device /*, dynamic_smem*/)( // kernel function add_constant_kernel, // kernel arguments static_cast(dst.data_ptr()), static_cast(src.data_ptr()), num_elements); } } // namespace sglang ``` ### STEP 2: Create Python Interfaces Next, expose the kernel through a Python wrapper. Create a new file at [kernels/ops/attention/add\_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/attention/add_constant.py) and expose the needed interfaces. ```python Example theme={null} from __future__ import annotations from typing import TYPE_CHECKING import torch from sglang.kernels.jit.utils import cache_once, load_jit, make_cpp_args if TYPE_CHECKING: from tvm_ffi.module import Module @cache_once def _jit_add_constant_module(constant: int) -> Module: args = make_cpp_args(constant) # pass all the template argument return load_jit( "add_constant", *args, cuda_files=["add_constant.cuh"], cuda_wrappers=[("add_constant", f"add_constant<{args}>")], ) def add_constant(src: torch.Tensor, constant: int) -> torch.Tensor: if not src.is_cuda: raise RuntimeError("src must be a CUDA tensor") if src.dtype != torch.int32: raise RuntimeError(f"Unsupported dtype {src.dtype}. Supported: int32") dst = torch.empty_like(src) module = _jit_add_constant_module(constant) module.add_constant(dst, src) return dst ``` Keep the Python wrapper thin, but still validate the basic invariants such as device and dtype before dispatch. In the current JIT/FFI path, invalid tensors are not always rejected safely before launch. ### STEP 3: Use your kernel Finally, import and use the kernel like a regular Python function: ```python Example theme={null} from sglang.kernels.jit.add_constant import add_constant ``` For a complete, runnable example, refer to [test\_add\_constant.py](https://github.com/sgl-project/sglang/blob/main/test/registered/jit/test_add_constant.py). ## C++ Include Library Reference The JIT kernel framework provides a set of reusable C++ headers in `python/sglang/kernels/jit/include/sgl_kernel/`. Each header is designed to be lightweight and self-contained. Below is a summary of each header and its key APIs. ### Core Utilities
Header Namespace Purpose
utils.h host Host-side essentials: RuntimeCheck, CHECK\_HOST(cond) \<\< ..., Panic, div\_ceil, irange
utils.cuh device / host Type aliases (fp16\_t, bf16\_t, ...), SGL\_DEVICE macro, PDL helpers, LaunchKernel, RuntimeDeviceCheck, CHECK\_CUDA(expr) \<\< ...
source\_location.h (global) Portable std::source\_location wrapper for error reporting
runtime.cuh host::runtime CUDA runtime queries: get\_blocks\_per\_sm, get\_sm\_count, get\_cc\_major, get\_runtime\_version, get\_available\_dynamic\_smem\_per\_block
### Tensor Validation
Header Namespace Purpose
tensor.h host TensorMatcher, SymbolicSize, SymbolicDType, SymbolicDevice
### Math & Type System
Header Namespace Purpose
math.cuh device::math max, min, abs, sqrt, rsqrt, exp, sin, cos, constants
type.cuh (global) / device DTypeTrait\, packed\_t\, device::cast\(from)
### Memory Access
Header Namespace Purpose
vec.cuh device AlignedVector\ - vectorized load/store (up to 128-bit; 256-bit requires Blackwell GPUs)
tile.cuh device::tile Memory\ - cooperative tiled memory I/O (thread/warp/CTA)
### Parallel Primitives
Header Namespace Purpose
warp.cuh device::warp reduce\ (SUM/MAX/MIN, grouped or inter-group) and reduce\_sum / reduce\_max / reduce\_min wrappers via \_\_shfl\_xor\_sync
cta.cuh device::cta reduce\_max across warps via shared memory
atomic.cuh device::atomic max - atomic float max (CUDA + ROCm fallback)
### Reusable Kernel Templates
Header Namespace Purpose
impl/norm.cuh host::norm / device::norm RMSNorm building blocks (warp & CTA paths, StorageType)
# Evaluating New Models with SGLang Source: https://docs.sglang.io/docs/developer_guide/evaluating_new_models This document provides commands for evaluating models' accuracy and performance. Before open-sourcing new models, we strongly suggest running these commands to verify whether the score matches your internal benchmark results. **For cross verification, please submit commands for installation, server launching, and benchmark running with all the scores and hardware requirements when open-sourcing your models.** [Reference: MiniMax M2](https://github.com/sgl-project/sglang/pull/12129) ## Accuracy ### LLMs SGLang provides built-in scripts to evaluate common benchmarks. **MMLU** ```bash Command theme={null} python -m sglang.test.run_eval \ --eval-name mmlu \ --port 30000 \ --num-examples 1000 \ --max-tokens 8192 ``` **GSM8K** ```bash Command theme={null} python -m sglang.test.run_eval \ --eval-name gsm8k \ --host 127.0.0.1 \ --port 30000 \ --num-examples 200 \ --num-shots 5 ``` **HellaSwag** ```bash Command theme={null} python benchmark/hellaswag/bench_sglang.py \ --host http://127.0.0.1 \ --port 30000 \ --num-questions 200 \ --num-shots 20 ``` **GPQA** ```bash Command theme={null} python -m sglang.test.run_eval \ --eval-name gpqa \ --port 30000 \ --num-examples 198 \ --max-tokens 120000 \ --repeat 8 ``` For reasoning models, add `--thinking-mode `. Supported values are `deepseek-v3`, `qwen-3`, `glm-45`, and `kimi-k2`. You may skip it if the model has forced thinking enabled. **HumanEval** ```bash Command theme={null} pip install human_eval python -m sglang.test.run_eval \ --eval-name humaneval \ --num-examples 10 \ --port 30000 ``` ### VLMs **MMMU** ```bash Command theme={null} python benchmark/mmmu/bench_sglang.py \ --port 30000 \ --concurrency 64 ``` You can set max tokens by passing `--extra-request-body '{"max_tokens": 4096}'`. For models capable of processing video, we recommend extending the evaluation to include `VideoMME`, `MVBench`, and other relevant benchmarks. ## Performance Performance benchmarks measure **Latency** (Time To First Token - TTFT) and **Throughput** (tokens/second). ### LLMs **Latency-Sensitive Benchmark** This simulates a scenario with low concurrency (e.g., single user) to measure latency. ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --num-prompts 10 \ --max-concurrency 1 ``` **Throughput-Sensitive Benchmark** This simulates a high-traffic scenario to measure maximum system throughput. ```bash Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --host 0.0.0.0 \ --port 30000 \ --dataset-name random \ --num-prompts 1000 \ --max-concurrency 100 ``` **Single Batch Performance** You can also benchmark the performance of processing a single batch offline. ```bash Command theme={null} python -m sglang.bench_one_batch_server \ --model \ --batch-size 8 \ --input-len 1024 \ --output-len 1024 ``` You can run more granular benchmarks: * **Low Concurrency**: `--num-prompts 10 --max-concurrency 1` * **Medium Concurrency**: `--num-prompts 80 --max-concurrency 16` * **High Concurrency**: `--num-prompts 500 --max-concurrency 100` ## Reporting Results For each evaluation, please report: 1. **Metric Score**: Accuracy % (LLMs and VLMs); Latency (ms) and Throughput (tok/s) (LLMs only). 2. **Environment settings**: GPU type/count, SGLang commit hash. 3. **Launch configuration**: Model path, TP size, and any special flags. 4. **Evaluation parameters**: Number of shots, examples, max tokens. # MSProbe Debugging Guide Source: https://docs.sglang.io/docs/developer_guide/msprobe_debugging_guide MSProbe is a debugging tool for AI models that diagnoses accuracy anomalies and numerical errors during model training and inference. It captures and monitors intermediate data (feature maps, weights, activations, layer outputs) and contextual metadata (prompts, tensor dtypes, hardware configuration), and supports visual analysis to systematically trace the root cause of accuracy degradation or numerical errors (e.g., NaN/Inf, output drift, mismatched predictions). ## Basic Details ### Background Concepts: MSProbe Dumping Levels MSProbe supports three accuracy levels for data dumping, each for different debugging needs: * **L0**: Dumps tensors/statistics at the **module level** and generates `construct.json` (for network structure reconstruction in visualization). Requires passing a model/submodule handle. * **L1**: Dumps tensors/statistics at the **torch API level**, suitable for fine-grained API-level numerical checking. * **mix**: Combines L0 + L1, ideal for scenarios that require both **graph reconstruction** and **numerical comparison**. ### Prerequisites: Install MSProbe Install MSProbe with pip: ```shell theme={null} pip install mindstudio-probe --pre ``` ### Key Configuration Parameters MSProbe uses a JSON configuration file for customized data dumping. All core parameters are listed in the table below, with the default JSON configuration provided for reference. #### Configuration Parameter Table | Field | Description | Required | | :----------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------: | | `task` | Type of dump task. Common PyTorch values include `"statistics"` and `"tensor"`. A statistics task collects tensor statistics (mean, variance, max, min, etc.) while a tensor task captures arbitrary tensors. | Yes | | `dump_path` | Directory where dump results are stored. When omitted, `MSProbe` uses its default path. | No | | `rank` | Ranks to sample. An empty list collects every rank. For single-card tasks you must set this field to `[]`. | No | | `step` | Token iteration(s) to sample. An empty list means every iteration. | No | | `level` | Dump level string (`"L0"`, `"L1"`, or `"mix"`). `L0` targets `nn.Module`, `L1` targets `torch.api`, and `mix` collects both. | Yes | | `async_dump` | Whether to enable asynchronous dump (supported for PyTorch `statistics`/`tensor` tasks). Defaults to `false`. | No | | `scope` | Customize the scope of dump. Provide two module or API names that follow the tool's naming convention to lock a range, only data between the two names will be dumped. An empty list dumps every module or torch API.

Examples:
`"scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"]`
`"scope": ["Tensor.add.0.forward", "Functional.square.2.forward"]`

The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`. | No | | `list` | Customize dump list, only dumps elements from the list. An empty list dumps every module or torch API. Options include:

•Supply the full names of specific APIs in PyTorch eager mode to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.backward"]`.
•When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`.
•Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded. | No | #### Default configuration ```json theme={null} { "task": "statistics", "dump_path": "./dump_path", "rank": [], "step": [], "level": "L1", "async_dump": false, "statistics": { "scope": [], "list": [], "data_mode": [ "all" ], "summary_mode": "statistics" }, "tensor": { "scope": [], "list": [], "data_mode": [ "all" ], "file_format": "npy" }, "acc_check": { "white_list": [], "black_list": [], "error_data_path": "./" } } ``` #### Outputs Dump files are written into `dump_path` you defined. They usually contain: * `dump.json`, which records metadata such as dtype, shape, min, max, mean, L2 norm, and `requires_grad`. * `construct.json`, hierarchical structure description, when `level` is `L0` or `mix` (required for visualization), its content is not empty. * `stack.json`, record the call stack information of API/Module. * `dump_tensor_data`, generated when `task` is `tensor` and save the collected tensor data. See [dump directory description](#dump-directory-description) for details. > **Note**: When MSProbe is enabled, cuda graph is disabled (disable\_cuda\_graph=True) because MSProbe only supports dump > in eager mode, warmup is disabled (skip\_server\_warmup=True) because there is no need to dump data for this stage. ## End-to-End Examples MSProbe’s full debugging workflow follows **Enable → Collect Data → Visualize → Analyze Root Cause**. Below is a common E2E example for SGLang-based model inference debugging. ### Example : Advanced Debugging with Custom Configuration Suitable for targeted debugging (e.g., only collect statistics data for specific ranks/steps, enable mix level for graph reconstruction + numerical comparison) and root cause analysis via **problem vs. benchmark comparison**. #### Step 1: Enable ##### Prepare Custom Configuration JSON Create `msprobe-config.json` (dump statistics data for rank0/1, step0/1, mix level): ```json theme={null} { "task": "statistics", "dump_path": "./problem_dump", "rank": [ 0, 1 ], "step": [ 0, 1 ], "level": "mix", "async_dump": false, "statistics": { "scope": [], "list": [], "data_mode": [ "all" ], "summary_mode": "statistics" } } ``` ##### Enable MSProbe with Custom Configuration in SGLang Launch the SGLang server and specify the configuration file path with `--msprobe-dump-config`: ```bash theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen2.5-0.5B-Instruct \ --host 127.0.0.1 \ --port 1027 \ --msprobe-dump-config /home/msprobe-config.json ``` #### Step 2: Collect Data ##### Collect Dump Data for Problem & Benchmark Sides Send normal inference requests to trigger model running (MSProbe automatically collects data during request processing): ```bash theme={null} curl -H "Content-type: application/json" \ -X POST \ -d '{ "model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [ { "role": "user", "content": "Hello, my name is" } ], "max_tokens": 10 }' \ http://127.0.0.1:1027/v1/chat/completions ``` * **Problem side**: Run the above SGLang server (with the accuracy/numerical issue) and send inference request; dump data is saved to `./problem_dump`. * **Benchmark side**: Launch a normal SGLang server (without the issue, e.g., stable framework version/operator) with the **same custom configuration** and send the **same inference request**; rename the dump directory to `./bench_dump`. > **Key Requirement**: Problem and benchmark dumps must use the same inputs and sampling points (rank/step) > for valid comparison. ##### Check Generated Dump Files Dump files are saved to `./problem_dump` and `./bench_dump` you defined and include core files for subsequent analysis: * `dump.json`: Records tensor metadata of APIs and modules (dtype, shape, min/max/mean, L2 norm, `requires_grad`, etc.). * `stack.json`: Logs call stack information of APIs and modules. * `construct.json`: hierarchical structure description, required for visualization, its content is not empty. #### Step 3: Visualize ##### Visualize Problem vs. Benchmark Comparison (Multi-Rank) Generate a multi-rank comparison visualization file (mix level generates `construct.json` for graph reconstruction): ```shell theme={null} msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output ``` * `-tp`: Path to problem-side dump data * `-gp`: Path to benchmark-side dump data * `-o`: Output directory for visualization files If you want overflow check (for NaN/Inf detection), please specify the parameter `-oc` ```shell theme={null} msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output -oc ``` After the comparison or build task finishes, a `compare_{timestamp}.vis.db` file is created under `graph_output`. The dumped data can be used to visualize and analyze differences using tables or charts generated with visualization tools such as Matplotlib and Excel. ##### Launch TensorBoard Start TensorBoard: ```bash theme={null} tensorboard --logdir ./graph_output --bind_all --port 6006 ``` #### Step 4: Analyze Root Cause ##### Locate Root Cause Root Cause Analysis in TensorBoard: * Divergent nodes (with accuracy/numerical differences) are highlighted in **red** (darker red = larger difference). * Click on divergent nodes to view detailed tensor data (inputs/outputs, parameters) and API/module call stacks. * Use the **search/filter** function to quickly locate key layers/APIs (e.g., "relu", "conv"). * Switch between ranks/steps via the UI to check cross-rank/cross-step divergence. * Check the **overflow check** tab for NaN/Inf values in specific nodes (the direct cause of numerical instability). ##### Verify the Root Cause After locating the divergent node (e.g., a specific Conv layer or torch API with abnormal tensor values), verify by: * Narrowing the dump scope to this node (via `scope`/`list` in the configuration file) for fine-grained data collection. * Modifying the problematic layer/API (e.g., replacing the operator, adjusting the dtype) and re-running the debugging workflow to confirm the issue is resolved. ## Troubleshooting ### No Dump Files Generated 1. To confirm if MSProbe is installed, use `pip show mindstudio-probe` to troubleshoot. If it is installed, the MSProbe version information will be printed. If it is confirmed that it has not been installed, please use `pip install mindstudio-probe --pre` for installation; 2. Confirm the `--msprobe-dump-config` parameter points to the **correct JSON file path**. ### Dump Files Are Too Large (Excessive Data) 1. Start with `task: "statistics"` instead of `"tensor"` to collect only tensor statistics (avoids raw tensor dump); 2. Narrow the dump range with the `scope` field (specify start/end module/API); 3. Filter dump targets with the `list` field (only dump specific modules/APIs or substrings); 4. Sample specific `rank` and `step` (avoid dumping all ranks/iterations). ### TensorBoard Visualization Fails 1. Confirm `construct.json` is not empty (requires `level: L0` or `mix` – L1 does not generate graph files); 2. Check that the `-tp` (problem dump) and `-gp` (benchmark dump) paths point to **valid rank/step subdirectories** ( e.g., `step0/rank0`); 3. Ensure the MSProbe version is up-to-date (reinstall with `pip install mindstudio-probe --pre --upgrade`); 4. Verify TensorBoard is installed and the `--logdir` parameter points to the directory containing `.vis.db` files (not the file itself). ### Numerical Comparison Shows No Divergence But Model Accuracy Is Low 1. Expand the dump `step` range (check more token iterations for late-stage divergence); 2. Switch to `task: "tensor"` (statistics may mask subtle numerical differences in raw tensor data); 3. Ensure the problem and benchmark dumps use **the same input data/hardware configuration** (different inputs lead to invalid comparisons); 4. Use the `manual mapping` feature in TensorBoard (automatic mapping may miss some nodes for custom models). *** ## Appendix ### Dump directory description ```text theme={null} ├── problem_dump or bench_dump │ ├── step0 │ │ ├── rank0 │ │ │ ├── dump_tensor_data │ │ │ │ ├── Tensor.permute.1.forward.pt │ │ │ │ ├── Functional.linear.5.backward.output.pt # Format: {api_type}.{api_name}.{call_count}.{forward/backward}.{input/output}.{arg_index}. │ │ │ │ │ # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument). │ │ │ │ ├── Module.conv1.Conv2d.forward.0.input.0.pt # Format: {Module}.{module_name}.{class_name}.{forward/backward}.{call_count}.{input/output}.{arg_index}. │ │ │ │ ├── Module.conv1.Conv2d.forward.0.parameters.bias.pt # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}. │ │ │ │ └── Module.conv1.Conv2d.parameters_grad.weight.pt # Module parameter gradients: {Module}.{module_name}.{class_name}.parameters_grad.{parameter_name}. Gradients do not include call_count because the same gradient updates all invocations. │ │ │ │ # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt. │ │ │ ├── dump.json │ │ │ ├── stack.json │ │ │ ├── dump_error_info.log │ │ │ └── construct.json │ │ ├── rank1 │ │ │ ├── dump_tensor_data │ │ │ │ └── ... │ │ │ ├── dump.json │ │ │ ├── stack.json │ │ │ ├── dump_error_info.log │ │ │ └── construct.json │ │ ├── ... │ │ │ │ │ └── rank7 │ ├── step1 │ │ ├── ... │ ├── step2 ``` * `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios the directory is simply named `rank`. * `dump_tensor_data`: Save the collected tensor data. * `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2 norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`. See [dump.json file description](#dump-json-file-description) for details. * `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log. * `stack.json`: Call stacks for APIs/modules. * `construct.json`: Hierarchical structure description. Empty when `level=L1`. ### dump.json file description #### L0 level An L0 `dump.json` contains forward/backward I/O for modules together with parameters and parameter gradients. Using PyTorch's `Conv2d` as an example, the network code looks like: `output = self.conv2(input) # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)` `dump.json` contains the following entries: * `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs` represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases. * `Module.conv2.Conv2d.parameters_grad`: Parameter gradients (weight and bias). * `Module.conv2.Conv2d.backward.0`: Backward data of the module. `input` represents gradients that flow into the module (gradients of the forward outputs) and `output` represents gradients that flow out (gradients of the module inputs). **Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`, module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`.
L0 dump.json ```json theme={null} { "task": "tensor", "level": "L0", "framework": "pytorch", "dump_data_dir": "/dump/path", "data": { "Module.conv2.Conv2d.forward.0": { "input_args": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 16, 14, 14 ], "Max": 1.638758659362793, "Min": 0.0, "Mean": 0.2544615864753723, "Norm": 70.50277709960938, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.input.0.pt" } ], "input_kwargs": {}, "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 32, 10, 10 ], "Max": 1.6815717220306396, "Min": -1.5120246410369873, "Mean": -0.025344856083393097, "Norm": 149.65576171875, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.output.0.pt" } ], "parameters": { "weight": { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 5, 5 ], "Max": 0.05992485210299492, "Min": -0.05999220535159111, "Mean": -0.0006165213999338448, "Norm": 3.421217441558838, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.parameters.weight.pt" }, "bias": { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32 ], "Max": 0.05744686722755432, "Min": -0.04894155263900757, "Mean": 0.006410328671336174, "Norm": 0.17263513803482056, "requires_grad": true, "data_name": "Module.conv2.Conv2d.forward.0.parameters.bias.pt" } } }, "Module.conv2.Conv2d.parameters_grad": { "weight": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 5, 5 ], "Max": 0.018550323322415352, "Min": -0.008627401664853096, "Mean": 0.0006675920449197292, "Norm": 0.26084786653518677, "requires_grad": false, "data_name": "Module.conv2.Conv2d.parameters_grad.weight.pt" } ], "bias": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32 ], "Max": 0.014914230443537235, "Min": -0.006656786892563105, "Mean": 0.002657240955159068, "Norm": 0.029451673850417137, "requires_grad": false, "data_name": "Module.conv2.Conv2d.parameters_grad.bias.pt" } ] }, "Module.conv2.Conv2d.backward.0": { "input": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 32, 10, 10 ], "Max": 0.0015069986693561077, "Min": -0.001139344065450132, "Mean": 3.3215508210560074e-06, "Norm": 0.020567523315548897, "requires_grad": false, "data_name": "Module.conv2.Conv2d.backward.0.input.0.pt" } ], "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 8, 16, 14, 14 ], "Max": 0.0007466732058674097, "Min": -0.00044813455315306783, "Mean": 6.814070275140693e-06, "Norm": 0.01474067009985447, "requires_grad": false, "data_name": "Module.conv2.Conv2d.backward.0.output.0.pt" } ] } } } ```
#### L1 level An L1 `dump.json` records forward/backward I/O for APIs. Using PyTorch's `relu` function as an example (`output = torch.nn.functional.relu(input)`), the file contains: * `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword inputs, and `output` stores the forward outputs. * `Functional.relu.0.backward`: Backward data of the API. `input` represents the gradients of the forward outputs, and `output` represents the gradients that flow back to the forward inputs.
L1 dump.json ```json theme={null} { "task": "tensor", "level": "L1", "framework": "pytorch", "dump_data_dir": "/dump/path", "data": { "Functional.relu.0.forward": { "input_args": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 1.3864083290100098, "Min": -1.3364859819412231, "Mean": 0.03711778670549393, "Norm": 236.20692443847656, "requires_grad": true, "data_name": "Functional.relu.0.forward.input.0.pt" } ], "input_kwargs": {}, "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 1.3864083290100098, "Min": 0.0, "Mean": 0.16849493980407715, "Norm": 175.23345947265625, "requires_grad": true, "data_name": "Functional.relu.0.forward.output.0.pt" } ] }, "Functional.relu.0.backward": { "input": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 0.0001815402356442064, "Min": -0.00013352684618439525, "Mean": 0.00011915402356442064, "Norm": 0.007598237134516239, "requires_grad": false, "data_name": "Functional.relu.0.backward.input.0.pt" } ], "output": [ { "type": "torch.Tensor", "dtype": "torch.float32", "shape": [ 32, 16, 28, 28 ], "Max": 0.0001815402356442064, "Min": -0.00012117840378778055, "Mean": 2.0098118724831693e-08, "Norm": 0.006532244384288788, "requires_grad": false, "data_name": "Functional.relu.0.backward.output.0.pt" } ] } } } ```
#### mix level A `mix` dump.json contains both L0 and L1 level data; the file format is the same as the examples above. # Developer Guide Source: https://docs.sglang.io/docs/developer_guide/overview Contributing to SGLang — development setup, benchmarking, and evaluation. * [Contribution Guide](./contribution_guide) * [Add an out-of-tree serve backend](/docs/developer_guide/serve_backend_plugins) * [Development Guide (Docker)](./development_guide_using_docker) * [JIT Kernels](./development_jit_kernel_guide) * [Quantization Contribution Guide](./quantization_contribution_guide) * [Benchmark and Profiling](./benchmark_and_profiling) * [Bench Serving](./bench_serving) * [Evaluating New Models](./evaluating_new_models) * [MSProbe Debugging Guide](./msprobe_debugging_guide) # Quantization Contribution Guide Source: https://docs.sglang.io/docs/developer_guide/quantization_contribution_guide This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels. ## Design Goals Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently. Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194): * **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme. * **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types. * **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends. Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization//` and split schemes into `schemes/`. ## Recommended File Layout Use this layout for a quantization method that has multiple schemes or backend-specific execution paths: ```text theme={null} python/sglang/srt/layers/quantization// __init__.py .py schemes/ __init__.py _scheme.py _linear.py _moe.py _.py ``` Backend kernels should live under the hardware backend they target: ```text theme={null} python/sglang/srt/hardware_backend/gpu/quantization/_kernels.py python/sglang/srt/hardware_backend/npu/quantization/_kernels.py ``` Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies. ## Adding or Refactoring a Quantization Method 1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed. 2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`. 3. Move layer-specific weight creation and weight loading into scheme classes. 4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules. 5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics. 6. Preserve compatibility for existing quantized checkpoints and runtime flags. 7. Add tests that cover both config parsing and execution paths touched by the change. For examples, see the AWQ and GPTQ refactors: * [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls. * [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ. ## Tests and Validation Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change. For Python-only structure changes: ```bash theme={null} ruff check git diff --check ``` For quantized model behavior: * Launch at least one representative model for each touched quantization method. * Send a `/generate` request and confirm the output path succeeds. * Run an accuracy sanity test if the change can affect numerics. * Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch. For backend-specific changes: * Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel). * Validate NPU changes on a supported Ascend environment. * Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description. ## PR Checklist Before requesting review, make sure the PR description includes: * The quantization method and backend paths changed. * The issue, design proposal, or roadmap item the PR follows. * Any compatibility notes for existing checkpoints or flags. * Accuracy results when model outputs can change. * Benchmark or profiling results when runtime performance can change. * The exact local checks and model launch tests that were run. Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details. # Add an out-of-tree serve backend Source: https://docs.sglang.io/docs/developer_guide/serve_backend_plugins Connect an ecosystem runtime to sglang serve through the versioned serve backend plugin API. A serve backend plugin connects an out-of-tree runtime to the SGLang-owned CLI: ```bash theme={null} sglang serve MODEL_PATH --model-type BACKEND_NAME [BACKEND_OPTIONS] ``` The extension retains its own argument parser, process topology, API endpoints, hardware requirements, and release cycle. The core CLI owns backend selection and common child-process cleanup. ## Prerequisites * Install your extension and a compatible SGLang version in the same Python 3.10+ environment. * Declare the SGLang version range tested by your extension in its package dependencies. * Keep the backend factory and detector independent of GPU initialization and model loading. The plugin API itself is platform-independent. Your backend defines its supported operating systems, accelerators, parallelism options, and authentication requirements. ## Keep one owner for the executable Only the `sglang` distribution should publish a console script named `sglang`. Your extension registers package metadata under `sglang.serve_backends`; it must not publish another `sglang` script. This prevents installation order from replacing the command and ensures uninstalling an extension does not remove the core executable. You can retain a project-specific executable as a compatibility alias: ```bash theme={null} my-runtime serve MODEL_PATH sglang serve MODEL_PATH --model-type my_runtime ``` Both commands should call the same backend implementation. ## Register a backend factory Add a zero-argument factory to your extension's `pyproject.toml`: ```toml theme={null} [project] name = "my-sglang-runtime" version = "0.1.0" dependencies = ["sglang"] [project.entry-points."sglang.serve_backends"] my_runtime = "my_sglang_runtime.sglang_backend:create_backend" ``` The entry point name, `my_runtime`, becomes an accepted `--model-type` value. Choose a distinctive name. `auto` and SGLang's in-tree backend names are reserved. ## Implement the backend Create `my_sglang_runtime/sglang_backend.py`: ```python theme={null} import argparse from sglang.cli.serve_backends import ( ServeBackend, ServeBackendDetection, ServeRequest, ) def detect(request: ServeRequest) -> ServeBackendDetection: if request.model_path is None: return ServeBackendDetection.UNKNOWN if supports_model(request.model_path): return ServeBackendDetection.MATCH return ServeBackendDetection.NO_MATCH def run(request: ServeRequest) -> None: parser = argparse.ArgumentParser(prog="sglang serve") parser.add_argument("--model-path", required=True) parser.add_argument("--pipeline-parallel", type=int, default=1) args, remaining = parser.parse_known_args(request.argv) launch_runtime(args, remaining) def create_backend() -> ServeBackend: return ServeBackend(api_version=1, run=run, detect=detect) ``` Replace `supports_model()` and `launch_runtime()` with your extension's lightweight metadata check and blocking server launcher. A real `run()` call should block for the server lifetime. It must also honor `-h` and `--help` without launching a server; `argparse` does this automatically. Among serve backend entry points, explicit selection imports only the selected provider. Automatic selection loads installed backend factories and invokes their detectors, so importing this module and calling `detect()` must not initialize accelerators, import model weights, or start workers. ## Handle forwarded arguments SGLang removes `--model-type` and normalizes a positional Hugging Face model ID or local model directory before dispatch. For example: ```bash theme={null} sglang serve org/model --model-type my_runtime --pipeline-parallel 2 ``` Your backend receives: ```python theme={null} ("--model-path", "org/model", "--pipeline-parallel", "2") ``` The selected backend owns all remaining argument parsing and validation. Target backend-specific help with: ```bash theme={null} sglang serve --model-type my_runtime --help ``` ## Support config-only runtimes SGLang requires a model path by default. If your runtime resolves its model and parallelism settings from a configuration file, disable that validation: ```python theme={null} def create_backend() -> ServeBackend: return ServeBackend( api_version=1, run=run, detect=detect, requires_model_path=False, ) ``` You can then accept commands such as: ```bash theme={null} sglang serve --model-type my_runtime --config pipeline.yaml ``` Config-only requests generally require explicit `--model-type` unless your detector can identify the backend from the remaining arguments. ## Understand automatic routing The default `--model-type auto` follows these rules: 1. Backends without a detector remain explicit-only. 2. One `MATCH` selects that backend. 3. Multiple matches fail and require an explicit `--model-type`. 4. `UNKNOWN` and detector failures do not claim the request. 5. No matches preserve the existing LLM fallback. The registry does not resolve overlap by package installation order or a hidden priority. A backend can opt out of automatic routing by omitting its detector: ```python theme={null} ServeBackend(api_version=1, run=run, requires_model_path=False) ``` ## Maintain compatibility Declare the API version implemented by your extension as a literal. Do not copy SGLang's current version constant at runtime; a fixed value lets a future SGLang release detect an older plugin contract. SGLang rejects incompatible, duplicate, and reserved backend registrations with an actionable error. The public extension contract consists of: * `ServeRequest`: normalized backend arguments and the optional model path * `ServeBackend`: the runner, optional detector, and model-path requirement * `ServeBackendDetection`: `MATCH`, `NO_MATCH`, or `UNKNOWN` Test explicit selection, backend-specific help, automatic detection, ambiguous models, and installation or removal alongside the core `sglang` package. # Installation Source: https://docs.sglang.io/docs/get-started/install Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options. You can install SGLang using one of the methods below. This page primarily applies to common NVIDIA GPU platforms. For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../hardware-platforms/amd_gpu), [Apple Metal](../hardware-platforms/apple_metal), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPU](../hardware-platforms/tpu), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu). Prerequisites: Python 3.10 or higher. ## Method 1: With pip or uv It is recommended to use uv for faster installation: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op. The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang uv pip install --force-reinstall torch==2.13.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129 uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/ uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps ``` ### Nightly builds To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` To install a nightly build under Cuda 12, swap the index to `cu129`: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu129/ sglang ``` ### Quick fixes to common problems * If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions: 1. Use `export CUDA_HOME=/usr/local/cuda-` to set the `CUDA_HOME` environment variable. 2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above. ## Method 2: From source ```bash Command theme={null} # Use the last release branch git clone -b v0.5.18 https://github.com/sgl-project/sglang.git cd sglang # Install the python packages pip install --upgrade pip pip install -e "python" ``` **Quick fixes to common problems** * If you want to develop SGLang, you can try the dev docker image. Please refer to [setup docker container](../developer_guide/development_guide_using_docker#setup-docker-container). The docker image is `lmsysorg/sglang:dev`. ## Method 3: Using docker The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker). Replace `` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens). `latest` and `dev` are **mutable** tags: `latest` always points at the newest stable release, while `dev` is rebuilt daily from the latest `main` and includes build/development tools. Because they are overwritten over time, pin an immutable version tag for reproducible deployments — e.g. `lmsysorg/sglang:v0.5.18`. Browse all released versions on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags). ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` For production deployments, use the `runtime` variant which is significantly smaller (\~40% reduction) by excluding build tools and development dependencies: ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest-runtime \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly). Notes: * SGLang is shipped with CUDA 13 environment by default. To run SGLang on CUDA 12 environment, please use images with `-cu12` or `-cu129` suffix, such as `lmsysorg/sglang:latest-cu129` or `lmsysorg/sglang:dev-cu12`. ## Method 4: Using Kubernetes Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs). 1. Option 1: For single node serving (typically when the model size fits into GPUs on one node) Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example. 2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as `DeepSeek-R1`) Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service. ## Method 5: Using docker compose > This method is recommended if you plan to serve it as a service. > A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml). 1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine 2. Execute the command `docker compose up -d` in your terminal. ## Method 6: Run on Kubernetes or Clouds with SkyPilot To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot). 1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html). 2. Deploy on your own infra with a single command and get the HTTP API endpoint: SkyPilot YAML: sglang.yaml}> ```yaml Config theme={null} # sglang.yaml envs: HF_TOKEN: null resources: image_id: docker:lmsysorg/sglang:latest accelerators: A100 ports: 30000 run: | conda deactivate python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --host 0.0.0.0 \ --port 30000 ``` ```bash Command theme={null} # Deploy on any cloud or Kubernetes cluster. Use --cloud to select a specific cloud provider. HF_TOKEN= sky launch -c sglang --env HF_TOKEN sglang.yaml # Get the HTTP API endpoint sky status --endpoint 30000 sglang ``` 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve). ## Method 7: Run on AWS SageMaker To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy) Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://aws.github.io/deep-learning-containers/reference/available_images/#sglang). To deploy a pre-built SGLang Deep Learning Container without building your own image, see [Amazon SageMaker AI](/docs/basic_usage/aws_sagemaker). To host a model with your own container, follow the following steps: 1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script. 2. Push your container onto AWS ECR. Dockerfile Build Script: build-and-push.sh}> ```bash Command theme={null} #!/bin/bash AWS_ACCOUNT="" AWS_REGION="" REPOSITORY_NAME="" IMAGE_TAG="" ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com" IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}" echo "Starting build and push process..." # Login to ECR echo "Logging into ECR..." aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY} # Build the image echo "Building Docker image..." docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile . echo "Pushing ${IMAGE_URI}" docker push ${IMAGE_URI} echo "Build and push completed successfully!" ``` 3. Deploy a model for serving on AWS Sagemaker, refer to [deploy\_and\_serve\_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk). 1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker. 2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`. 3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli. 4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`. ## Common Notes * [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub. * To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`. # Quickstart Source: https://docs.sglang.io/docs/get-started/quickstart Get up and running with SGLang in minutes: install, launch a server, and send your first request. ## Overview This guide walks you through the entire flow of getting started with SGLang: 1. **Install** SGLang 2. **Launch** an inference server 3. **Send requests** using cURL, OpenAI Python client, Python `requests`, or the native SGLang API By the end, you'll have a working SGLang server responding to your prompts. *** ## Prerequisites * **Python**: 3.10 or higher * **GPU**: NVIDIA GPU with CUDA support (sm80 and above, e.g., A10, A100, L4, L40S, H100) * **OS**: Linux (recommended) For other platforms, see the dedicated guides for [AMD GPUs](../hardware-platforms/amd_gpu), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPUs](../hardware-platforms/tpu), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu). *** ## Installation We recommend using **uv** for faster installation: ```bash theme={null} pip install --upgrade pip pip install uv uv pip install --prerelease=allow sglang ``` ```bash theme={null} # Clone and install from source git clone https://github.com/sgl-project/sglang.git cd sglang pip install --upgrade pip pip install -e "python" ``` The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags). Replace `` with your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens): ```bash theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` For production deployments, use the smaller **runtime** variant (\~40% size reduction): ```bash theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest-runtime \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` If you encounter `OSError: CUDA_HOME environment variable is not set`, set it with: ```bash theme={null} export CUDA_HOME=/usr/local/cuda- ``` *** ## Launch a Server Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example: ```bash theme={null} python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000 ``` Wait until you see `The server is fired up and ready to roll!` in the terminal output. Once the server is running, API documentation is available at: * **Swagger UI**: `http://localhost:30000/docs` * **ReDoc**: `http://localhost:30000/redoc` * **OpenAPI Spec**: `http://localhost:30000/openapi.json` The server automatically applies the chat template from the Hugging Face tokenizer. You can override it with `--chat-template` when launching. *** ## Send Requests SGLang is fully **OpenAI API-compatible**, so you can use the same tools and libraries you already know. ### Using cURL ```bash theme={null} curl http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen/qwen2.5-0.5b-instruct", "messages": [ {"role": "user", "content": "What is the capital of France?"} ] }' ``` ### Using OpenAI Python Client Install the OpenAI Python library if you haven't: ```bash theme={null} pip install openai ``` Then send a request: ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, ) print(response.choices[0].message.content) ``` #### Streaming ```python Example theme={null} import openai client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") response = client.chat.completions.create( model="qwen/qwen2.5-0.5b-instruct", messages=[ {"role": "user", "content": "List 3 countries and their capitals."}, ], temperature=0, max_tokens=64, stream=True, ) for chunk in response: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` ### Using Python Requests ```python Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "qwen/qwen2.5-0.5b-instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}], } response = requests.post(url, json=data) print(response.json()) ``` ### Using the Native `/generate` API SGLang also provides a native `/generate` endpoint for more flexibility. ```python Example theme={null} import requests response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, }, ) print(response.json()) ``` #### Streaming with `/generate` ```python Example theme={null} import requests import json response = requests.post( "http://localhost:30000/generate", json={ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 32, }, "stream": True, }, stream=True, ) prev = 0 for chunk in response.iter_lines(decode_unicode=False): chunk = chunk.decode("utf-8") if chunk and chunk.startswith("data:"): if chunk == "data: [DONE]": break data = json.loads(chunk[5:].strip("\n")) output = data["text"] print(output[prev:], end="", flush=True) prev = len(output) ``` *** ## Offline Batch Inference (No Server) SGLang also supports offline batch inference using the `Engine` class directly -- no HTTP server required. ```python Example theme={null} import sglang as sgl llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct") prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] sampling_params = {"temperature": 0.8, "top_p": 0.95} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}\nGenerated text: {output['text']}\n") llm.shutdown() ``` *** ## Common Troubleshooting Set the `CUDA_HOME` environment variable to your CUDA install root: ```bash theme={null} export CUDA_HOME=/usr/local/cuda- ``` Switch to alternative backends by adding these flags when launching the server: ```bash theme={null} --attention-backend triton --sampling-backend pytorch ``` ```bash theme={null} pip3 install --upgrade flashinfer-python --force-reinstall --no-deps rm -rf ~/.cache/flashinfer ``` ```bash theme={null} export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas ``` *** # AMD GPUs Source: https://docs.sglang.io/docs/hardware-platforms/amd_gpu This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## System Configuration When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning: * [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) * [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html) * [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html) * [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html) * [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html) **NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system. Below are a few key settings to confirm or enable for SGLang: ### Update GRUB Settings In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`: ```text GRUB Configuration theme={null} pci=realloc=off iommu=pt ``` Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot. ### Disable NUMA Auto-Balancing ```bash Disable NUMA theme={null} sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing' ``` You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh). Again, please go through the entire documentation to confirm your system is using the recommended configuration. ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source ```bash Command theme={null} # Use the last release branch git clone -b v0.5.18 https://github.com/sgl-project/sglang.git cd sglang # Compile sgl-kernel pip install --upgrade pip cd python/sglang/kernels/aot python setup_rocm.py install # Install sglang python package along with diffusion support cd ../../../.. rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml pip install -e "python[all_hip]" ``` ### Install Using Docker (Recommended) The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker). The steps below show how to build and use an image. 1. Build the docker image. If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below. ```bash Command theme={null} docker build -t sglang_image -f rocm.Dockerfile . ``` 2. Create a convenient alias. ```bash Command theme={null} alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \ --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \ --security-opt seccomp=unconfined \ -v $HOME/dockerx:/dockerx \ -v /data:/data' ``` If you are using RDMA, please note that: * `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them. * You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`. 3. Launch the server. **NOTE:** Replace `` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path NousResearch/Meta-Llama-3.1-8B \ --host 0.0.0.0 \ --port 30000 ``` 4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine. ```bash Command theme={null} drun sglang_image \ python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 4000 \ --random-input 128 \ --random-output 128 ``` With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities. ## Quantization on AMD GPUs The [Quantization documentation](../advanced_features/quantization#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit\_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not. A few things to keep in mind: * FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box. * AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available. * MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`. * `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models. * `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4. Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with: ```bash Command theme={null} export SGLANG_USE_AITER=1 ``` Example -- serving an AWQ model: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \ --trust-remote-code \ --port 30000 --host 0.0.0.0 ``` Example -- FP8 online quantization: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ --quantization fp8 \ --port 30000 --host 0.0.0.0 ``` ## Examples ### Running DeepSeek-V3 The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command: ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --ipc=host \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V3 \ # <- here --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` [Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference. ### Running Llama3.1 Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command: ```bash Command theme={null} drun -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --ipc=host \ --env "HF_TOKEN=" \ sglang_image \ python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here --tp 8 \ --trust-remote-code \ --host 0.0.0.0 \ --port 30000 ``` ### Warmup Step When the server displays `The server is fired up and ready to roll!`, it means the startup is successful. # Apple Silicon with Metal Source: https://docs.sglang.io/docs/hardware-platforms/apple_metal This document describes how to run the SGLang serving runtime on Apple Silicon using [MLX](https://opensource.apple.com/projects/mlx/). SGLang Diffusion uses PyTorch MPS instead; see its [installation guide](/docs/sglang-diffusion/installation#platform-specific-apple-mps). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## Prerequisites The MLX runtime requires Apple Silicon with macOS 14 or newer, stable PyTorch 2.13.x, and stable MLX 0.32.0 or newer. The `srt_mps` extra installs PyTorch 2.13.0 and MLX 0.32.0 or newer; startup accepts stable PyTorch 2.13 patch releases and newer stable MLX releases. With `SGLANG_USE_MLX=1`, SGLang validates both framework versions and Metal availability during argument initialization and stops before resolving or downloading a model when the runtime is incompatible. Building the optional native Metal kernels in `sgl-kernel` requires the Metal shader compiler from the full Xcode application. The standalone Xcode Command Line Tools are not sufficient. After installing Xcode, select it with: ```bash theme={null} sudo xcode-select -s /Applications/Xcode.app/Contents/Developer ``` Verify the compiler with `xcrun -sdk macosx metal --version`. ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source ```bash theme={null} # Use the default branch git clone https://github.com/sgl-project/sglang.git cd sglang # Create and activate a virtual environment uv venv -p 3.12 sglang-metal source sglang-metal/bin/activate # (Optional) Compile sgl-kernel uv pip install --upgrade pip uv run python/sglang/kernels/aot/setup_metal.py install # Install sglang python package along with diffusion support rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml uv pip install -e "python[all_mps]" ``` ## Launch of the Serving Engine Launch the server with: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model \ --disable-cuda-graph \ --host 0.0.0.0 ``` **Key Parameters Explained:** 1. `SGLANG_USE_MLX=1` - Enables the use of MLX as the SGLang runtime backend (if disabled, SGLang will fall back to `torch.mps`, which has less support) 2. `--disable-cuda-graph` - Disables usage of CUDA graph, which is not relevant for Apple Metal. 3. `--disable-overlap-schedule` - Disables overlap scheduling (enabled/not present by default) achieved using MLX's `async_eval()` 4. `SGLANG_MLX_USE_CUSTOM_ROPE=1` - Enables the optional custom Metal RoPE kernel. It is disabled by default, so the MLX backend uses the standard RoPE path unless you opt in for A/B testing. 5. `SGLANG_MLX_FUSE_SWIGLU=1` - Enables the use of fused Swish-Gated Linear Unit kernel (disabled by default) 6. `SGLANG_MLX_CLEAR_CACHE_STEPS=256` - Sets the number of decode steps before clearing the MLX cache (256 by default) ## Quantization The MLX backend supports two quantization paths on Apple Silicon: 1. **Pre-quantized HF repos.** Any `mlx-community/-4bit` (or `-8bit`) repo loads directly through `mlx_lm.load(...)` — no extra flag needed. ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model-path mlx-community/Qwen3-0.6B-4bit \ --disable-cuda-graph ``` 2. **On-the-fly quantization.** For any fp16 model, pass `--quantization mlx_q4` or `--quantization mlx_q8` to have sglang quantize the weights at load time via `mlx_lm.utils.quantize_model` (group size 64, the mlx-community default). The quantized weights stay in process memory; the on-disk model is untouched. ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B \ --quantization mlx_q4 \ --disable-cuda-graph ``` Expected log line: ``` Quantizing MLX model on-the-fly: bits=4 group_size=64 (preset=mlx_q4) Quantization complete in 0.13s — active mem: 1.11 GB -> 0.31 GB (71.9% reduction) ``` The MLX backend silently ignores `--quantization mlx_q4` when the model is already quantized in its HF config (path 1), so the same flag is safe to pass either way. ## Benchmarking with Requests `sglang.benchmark.one_batch` calls the synchronous prefill/decode methods directly without going through the scheduler and the overlap code path. `sglang.benchmark.offline_throughput` can toggle overlap scheduling as it uses the scheduler and the overlap code path by using the flag `--disable-overlap-schedule`. ### Throughput Testing Basic synchronous one batch throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.one_batch \ --model-path \ --disable-cuda-graph \ --tp-size 1 \ --batch-size 1 \ --input-len 60 \ --output-len 10 ``` Synchronous offline throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \ --model-path \ --disable-cuda-graph \ --num-prompts 1 \ --disable-overlap-schedule ``` Asynchronous offline throughput: ```bash theme={null} SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \ --model-path \ --disable-cuda-graph \ --num-prompts 1 ``` # Contribution Guide Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/contribution_guide Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process. ## Install SGLang from Source ### Prepare Environment Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../getting-started/installation) to install the necessary dependencies. We recommend [using docker](../getting-started/installation#method-2-using-docker-image) to build the environment. ### Fork and clone the repository **Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally. ```bash theme={null} git clone https://github.com//sglang.git # if you are using docker, the environment is already set up. cd sglang export PYTHONPATH=$PWD/python:$PYTHONPATH ``` ## Format code with pre-commit We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run: ```bash theme={null} pip3 install pre-commit pre-commit install pre-commit run --all-files ``` * **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. * **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. * Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits. * To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`. ## Run and add tests All NPU tests are end-to-end (E2E) and require launching a server with real model weights. Tests live under [`test/registered/npu/`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu), organized by model type and functionality: ``` ascend/ ├── llm_models/ # Per-model inference accuracy ├── vlm_models/ # Vision-language models ├── embedding_models/ # Embedding model tests ├── rerank_models/ # Reranker model tests ├── reward_models/ # Reward model tests ├── interface/ # API correctness, function calling ├── basic_function/ # Cache, sampling, quantization, etc. └── test_npu_memory_consumption.py ``` ### Adding a test See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps: 1. Place your test file in the appropriate directory under `test/registered/npu/`. 2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support. 3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`. 4. Register your test with `register_npu_ci()`: ```python theme={null} from sglang.test.ci.ci_register import register_npu_ci register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a3", nightly=False) register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True) ``` ### Running tests locally ```bash theme={null} pytest test/registered/npu/llm_models/test_npu_qwen3_0_6b.py -v ``` For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). ### Register models for CI If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps: 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). 2. Make sure your model is pre-cached on the CI server and is at the path "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models//". If this is not the case, use following command on CI server: ```bash theme={null} modelscope download \ --model {your_model_repo}/{your_model} \ --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} ``` > Note: If you don’t have access to CI server, please ask maintainers ([zl19940307@163.com](mailto:zl19940307@163.com)) to download your model. 3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path). ## Write documentation We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase. For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md). ## Test the accuracy If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K. ```bash theme={null} # Launch a server python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct # Evaluate. --base-url must point at the server launched above. # The default SGLang server port is 30000; change it if you launched # the server with a different --port. python3 -m sglang.test.run_eval --base-url http://localhost:30000 --eval-name gsm8k --num-examples 200 ``` Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine. Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test. GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests. You can find additional accuracy eval examples in: * [test\_eval\_accuracy\_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py) * [test\_gpt\_oss\_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py) ## Benchmark the speed Refer to [Benchmark and Profiling](../../../developer_guide/benchmark_and_profiling). ## Requesting a review for merge You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. Then your PR can be merged. ## How to Trigger CI Tests We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) **PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`. For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: * `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). * `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**. * `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). * `/rerun-test [ ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `` is pytest-style `::[.]` (the `::TestClass` and `.` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once). If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). If you don’t have permission, please ask maintainers to trigger CI for you. ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: ```yaml theme={null} cool-down-minutes: description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting" type: number default: 120 ``` Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. ## Code style guidance * Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. * Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. * Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible. * Make functions as pure as possible. Avoid in-place modification of arguments. * Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`) * In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. * Keep tests run fast. * If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). * If a single job in a github workflow runs longer than 30 min, split it into smaller jobs/steps. * Reuse server launches in your unit tests to make tests run faster. * Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. * When supporting new hardware or features, follow these guidelines: * Do not drastically change existing code. * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`). * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. ## How to update sgl-kernel-npu Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./operator_development). ### Multi-PR workflow Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow: 1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass. 2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week). 3. **Reference the new version in SGLang**: * Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag. * Use the new operator in SGLang code. ## Tips for newcomers If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out the following materials as startup guide: * [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang. * [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. * [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance. If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). Thank you for your interest in SGLang. Happy coding! # Operator Development Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_development How to develop custom operators (Ascend C / Triton) for Ascend NPU and integrate them into the SGLang inference engine. ## Overview [SGL-Kernel-NPU](https://github.com/sgl-project/sgl-kernel-npu) is the official operator library provided by the SGLang framework for Ascend NPU. It includes two types of operator implementations: 1. **Ascend C operators**: High-performance C++ kernels written in Ascend C, compiled into `libsgl_kernel_npu.so`, and registered through PyTorch's custom operator mechanism (`TORCH_LIBRARY_FRAGMENT`). Called in SGLang via `torch.ops.npu.()`. 2. **Triton operators**: Python kernels written in Triton, adapted for Ascend NPU. Called directly via `from sgl_kernel_npu.xxx import ...`. When SGLang detects an NPU device, it automatically loads `sgl_kernel_npu` and uses its operators in place of GPU counterparts, providing optimized inference on Ascend hardware. ## Directory Structure The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g., `retrive_index`, `retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency. ```text theme={null} sgl-kernel-npu/ ├── csrc/ # Ascend C operator C++ sources │ ├── CMakeLists.txt # Build configuration │ ├── pytorch_extensions.cpp # PyTorch op registration (core integration file) │ └── / # One directory per operator │ ├── op_host/ # Host-side code (validation, tiling, launch) │ │ ├── .cpp │ │ └── tiling/ # Optional: tiling data │ └── op_kernel/ # Device-side code (Ascend C kernel on AICore) │ └── _kernel.cpp ├── include/ │ └── sgl_kenel_npu_ops.h # C++ interface declarations ├── python/ │ └── sgl_kernel_npu/ │ └── sgl_kernel_npu/ │ ├── __init__.py # Loads libsgl_kernel_npu.so │ ├── attention/ # Triton attention kernels │ ├── norm/ # Triton normalization kernels │ ├── activation/ # Triton activation kernels │ ├── fla/ # Triton linear attention kernels │ ├── mamba/ # Triton Mamba kernels │ ├── moe/ # Triton MoE kernels │ └── sample/ # Triton speculative decoding kernels ├── tests/ │ └── python/sgl_kernel_npu/ # One test file per operator ├── build.sh # Build script └── CMakeLists.txt # Root CMake configuration ``` ## Developing Ascend C Operators A complete Ascend C operator consists of two parts: * **Device part**: Kernel code running on the NPU AICore, responsible for actual computation. Written using the Ascend C API. * **Host part**: Code running on the CPU, responsible for parameter validation, data pre-processing, tiling, and kernel launch. We recommend starting with the [helloworld](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld) example, a simple operator that performs element-wise addition on two tensors. ### Step 1: Create the operator directory and files Create a new operator directory under `csrc/`, following the `op_host/` + `op_kernel/` structure: ```text theme={null} csrc// ├── op_host/ │ └── .cpp └── op_kernel/ └── _kernel.cpp ``` ### Step 2: Write the Device-side Kernel (op\_kernel) Device-side code runs on AICore and follows the Ascend C programming model. The core structure is a class with `Init()` and `Process()` methods, plus an `extern "C"` entry function. Using helloworld as an example: ```cpp theme={null} // csrc/helloworld/op_kernel/kernel_helloworld.cpp #include "kernel_operator.h" constexpr int32_t BUFFER_NUM = 2; class KernalHelloworld { public: __aicore__ inline KernalHelloworld() {} __aicore__ inline void Init(GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength) { // Compute workload for current block this->blockLength = totalLength / AscendC::GetBlockNum(); this->tileNum = 8; this->tileLength = this->blockLength / this->tileNum / BUFFER_NUM; // Set global memory buffers xGm.SetGlobalBuffer((__gm__ half *)x + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); yGm.SetGlobalBuffer((__gm__ half *)y + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); zGm.SetGlobalBuffer((__gm__ half *)z + this->blockLength * AscendC::GetBlockIdx(), this->blockLength); // Initialize pipeline queues pipe.InitBuffer(inQueueX, BUFFER_NUM, this->tileLength * sizeof(half)); pipe.InitBuffer(inQueueY, BUFFER_NUM, this->tileLength * sizeof(half)); pipe.InitBuffer(outQueueZ, BUFFER_NUM, this->tileLength * sizeof(half)); } __aicore__ inline void Process() { int32_t loopCount = this->tileNum * BUFFER_NUM; for (int32_t i = 0; i < loopCount; i++) { CopyIn(i); // Move data from Global Memory to Local Memory Compute(i); // Compute on Local Memory CopyOut(i); // Move results back to Global Memory } } private: __aicore__ inline void CopyIn(int32_t progress) { /* data copy-in... */ } __aicore__ inline void Compute(int32_t progress) { /* core computation... */ } __aicore__ inline void CopyOut(int32_t progress) { /* data copy-out... */ } private: AscendC::TPipe pipe; AscendC::TQue inQueueX, inQueueY; AscendC::TQue outQueueZ; AscendC::GlobalTensor xGm, yGm, zGm; uint32_t blockLength, tileNum, tileLength; }; // Entry function: the compile tool auto-generates aclrtlaunch_.h from this name extern "C" __global__ __aicore__ void helloworld( GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength) { KernalHelloworld op; op.Init(x, y, z, totalLength); op.Process(); } ``` **Key points:** * Class methods must be marked with `__aicore__`, indicating they run on AICore. * Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data movement and computation. * The entry function must be declared `extern "C" __global__ __aicore__`. The compile tool generates a host-callable launch header `aclrtlaunch_.h` from the function name. * Simple operators (e.g., helloworld, cache\_assign, lora) do not need extra workspace memory. Complex operators (e.g., mla\_preprocess, alloc\_extend, build\_tree) require temporary workspace memory and are compiled separately in `CMakeLists.txt`. For more in-depth Ascend C programming knowledge, refer to the [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html). ### Step 3: Write the Host-side Code (op\_host) Host-side code is responsible for passing PyTorch Tensors to the kernel and launching it. The key macro is `EXEC_KERNEL_CMD` (located in `csrc/utils/torch_helper.h`). ```cpp theme={null} // csrc/helloworld/op_host/helloworld.cpp #include "defines.h" // Provides HOST_API macro #include "torch_helper.h" // Provides EXEC_KERNEL_CMD macro #include "aclrtlaunch_helloworld.h" // Auto-generated by compile tool namespace sglang { namespace npu_kernel { HOST_API at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y) { // Create output tensor at::Tensor z = at::empty_like(x); // Define block count uint32_t blockDim = 8; // Compute total element count uint32_t totalLength = 1; for (uint32_t size : x.sizes()) { totalLength *= size; } // Launch kernel via EXEC_KERNEL_CMD macro EXEC_KERNEL_CMD(helloworld, blockDim, x, y, z, totalLength); return z; } } // namespace npu_kernel } // namespace sglang ``` **Key points:** * The namespace must be `sglang::npu_kernel`. * Function signatures follow the pattern `at::Tensor (const at::Tensor &input, ...)`. * For operators with multiple outputs, use `std::tuple` or non-const reference parameters. ### Step 4: Declare the C++ Interface (include/sgl\_kenel\_npu\_ops.h) Add the operator function declaration in `include/sgl_kenel_npu_ops.h`: ```cpp theme={null} // include/sgl_kenel_npu_ops.h namespace sglang { namespace npu_kernel { at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y); } // namespace npu_kernel } // namespace sglang ``` ### Step 5: Register the PyTorch Custom Operator (pytorch\_extensions.cpp) Register the operator in `csrc/pytorch_extensions.cpp` in two steps: define the schema and bind the implementation. ```cpp theme={null} // csrc/pytorch_extensions.cpp namespace { // 1. Define operator schema (used by torch.compile, etc.) TORCH_LIBRARY_FRAGMENT(npu, m) { m.def("helloworld(Tensor x, Tensor y) -> Tensor"); // ... other operator schemas ... } // 2. Bind implementation for the PrivateUse1 device (i.e., NPU) TORCH_LIBRARY_IMPL(npu, PrivateUse1, m) { m.impl("helloworld", TORCH_FN(sglang::npu_kernel::helloworld)); // ... other operator implementations ... } } // namespace ``` **Schema conventions:** * The namespace is fixed to `npu`. In SGLang, operators are called via `torch.ops.npu.()`. * Output tensor parameters use the `Tensor(a!)` mutating annotation. * Optional parameters use the `Tensor?` annotation, with `c10::optional` handling in the impl. * For detailed schema syntax, see the [PyTorch Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func). **Implementation binding rules:** * The device name is fixed to `PrivateUse1` (PyTorch NPU backend identifier). * Use the `TORCH_FN` macro to bind to the implementation function. * For complex operators with optional parameters, use lambda expressions to unpack the arguments. ### Step 6: Update the Build Configuration (csrc/CMakeLists.txt) Add the new operator's source files to `csrc/CMakeLists.txt`: **For operators not requiring workspace** (simple operators), add kernel source to `no_workspace_kernel`: ```cmake theme={null} ascendc_library(no_workspace_kernel STATIC # ... existing kernel files ... ${PROJECT_OP_SRC_BASE}//op_kernel/_kernel.cpp ) ``` **For operators requiring workspace** (complex operators), add kernel source to `workspace_kernel` with the `-DHAVE_WORKSPACE -DHAVE_TILING` compile flags: ```cmake theme={null} ascendc_library(workspace_kernel STATIC # ... existing kernel files ... ${PROJECT_OP_SRC_BASE}//op_kernel/_kernel.cpp ) ``` **Add host source files to `OP_SRCS`:** ```cmake theme={null} FILE(GLOB OP_SRCS # ... existing host files ... ${PROJECT_OP_SRC_BASE}//op_host/.cpp ) ``` ### Step 7: Build Build following the steps in the [python/sgl\_kernel\_npu/README.md](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md): ```bash theme={null} cd sgl-kernel-npu # Build all modules bash build.sh # Install the sgl_kernel_npu wheel pip install output/sgl_kernel_npu*.whl ``` The compiled `libsgl_kernel_npu.so` is copied into `python/sgl_kernel_npu/sgl_kernel_npu/lib/` and loaded by the Python package. ## Developing Triton Operators Triton operators are located under `python/sgl_kernel_npu/sgl_kernel_npu/`, organized by function category: ```text theme={null} python/sgl_kernel_npu/sgl_kernel_npu/ ├── attention/ # Attention (decode_attention, sinks_attention) ├── norm/ # Normalization (rmsnorm, fused_qk_norm, l1_norm) ├── activation/ # Activation (swiglu_oai, swiglu_quant) ├── fla/ # Linear attention (chunk, cumsum, wy_fast) ├── mamba/ # Mamba-related (causal_conv1d, state_update) ├── moe/ # MoE-related (mul_add, zero_experts) └── sample/ # Speculative decoding (verify_tree_greedy) ``` **Development steps:** 1. Create a new `.py` file in the appropriate category subdirectory. 2. Write the kernel using the Triton language, using existing operators in the same directory as templates. 3. Export functions in the corresponding `__init__.py` if needed. 4. Write tests under `tests/python/sgl_kernel_npu/`. **Note:** Many Triton operators are adapted from SGLang's GPU Triton kernels (e.g., comments in `fla/utils.py` note the original source). Pay special attention to differences between NPU and GPU when adapting. ## Integrating Operators into SGLang ### Ascend C Operator Integration After completing the [Steps 1-7](#developing-ascend-c-operators) above (writing the kernel, registering the torch op, building), and installing the `sgl-kernel-npu` wheel, call the operator in SGLang as follows: ```python theme={null} import sgl_kernel_npu # Loading the library auto-triggers libsgl_kernel_npu.so loading # Call the operator result = torch.ops.npu.helloworld(x, y) ``` Real-world usage in SGLang (from `sglang/srt/speculative/eagle_utils.py`): ```python theme={null} torch.ops.npu.build_tree_kernel_efficient( parent_list, selected_index, verified_seq_len, tree_mask, positions, retrive_index, retrive_next_token, retrive_next_sibling, topk, depth, draft_token_num, tree_mask_mode ) ``` ### Triton Operator Integration Import and call directly via Python: ```python theme={null} from sgl_kernel_npu.attention.decode_attention import decode_attention_fwd from sgl_kernel_npu.norm.rmsnorm_bias import rmsnorm_bias from sgl_kernel_npu.mamba.causal_conv1d import causal_conv1d_fwd # Direct function call output = decode_attention_fwd(q, k, v, ...) ``` Real-world usage in SGLang (from `sglang/srt/models/llama.py`): ```python theme={null} from sgl_kernel_npu.norm.split_qkv_rmsnorm_rope import split_qkv_rmsnorm_rope ``` ### sgl-kernel-npu Wheel Update Process Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow: 1. **Submit sgl-kernel-npu PR**: Add/modify operators in the sgl-kernel-npu repository, ensuring all tests pass. 2. **Bump sgl-kernel-npu version**: Update the version number in sgl-kernel-npu. Merging triggers an automatic PyPI release. 3. **Reference the new version in SGLang**: * Update the `sgl-kernel-npu` version requirement in SGLang's `python/pyproject.toml`. * Use the new operator in SGLang code. If not urgent, you can wait for a regular release (typically within one week). ## Writing Unit Tests Each operator needs a corresponding unit test under `tests/python/sgl_kernel_npu/`, using Python's `unittest` framework. Test file naming convention: `test_.py` ```python theme={null} # tests/python/sgl_kernel_npu/test_helloworld.py import unittest import torch import sgl_kernel_npu class TestHelloworld(unittest.TestCase): def test_helloworld_basic(self): x = torch.randn(2048, dtype=torch.float16, device="npu") y = torch.randn(2048, dtype=torch.float16, device="npu") z = torch.ops.npu.helloworld(x, y) expected = x + y torch.testing.assert_close(z, expected) if __name__ == "__main__": unittest.main() ``` Run tests: ```bash theme={null} python tests/python/sgl_kernel_npu/test_helloworld.py ``` **Testing checklist:** * Cover typical input shapes (power-of-2 sizes and non-standard sizes). * Cover different data types (bf16 / fp16, etc.). * For operators with in-place behavior, verify correctness of output tensors. * Compare against PyTorch native computation to verify accuracy. ## Code Style ### Pre-commit Checks sgl-kernel-npu uses pre-commit for consistent code style: ```bash theme={null} pip3 install pre-commit cd sgl-kernel-npu pre-commit install pre-commit run --all-files ``` **Note:** If `pre-commit run --all-files` fails the first time, run it again to ensure all lint errors are auto-fixed. All code must pass checks before submitting a PR. ### C++ Code Style * Use the C++17 standard. * Place all operator implementations under the `sglang::npu_kernel` namespace. * Follow existing code style; format using `.clang-format`. * Use `TORCH_CHECK` for error checking (not the standard GE `OP_ADD` macros). * Do not include unnecessary GE registration code (e.g., `OP_ADD()` macros). ### Python Code Style * Follow PEP 8. * Use snake\_case for file and function names. * When adapting from SGLang GPU code, note the original source in the file header. ### General Principles * Avoid code duplication: extract shared functions for any repeated code blocks over 5 lines. * Minimize device synchronization: reduce CPU-NPU sync operations like `tensor.item()` or `tensor.cpu()`. * Keep functions pure: avoid in-place argument modification. * Keep files concise: split files exceeding 2,000 lines. ## Submitting a PR 1. **Fork the repo**: Fork [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) on GitHub, then clone locally. 2. **Create a branch**: Create a new branch from `main`, e.g., `feature/add-my-op`. 3. **Develop and test**: Develop the operator and write tests following the steps above. Ensure all tests pass. 4. **Run pre-commit**: Ensure code formatting compliance. 5. **Commit and push**: ```bash theme={null} git add . git commit -m "feat: add operator" git push origin feature/add-my-op ``` 6. **Create a PR**: Open a Pull Request on GitHub from your branch to `sgl-project/sgl-kernel-npu:main`. 7. **Wait for CI and review**: CI checks include linting, compilation, and operator tests. After passing, wait for maintainer review and merge. ## References * [SGL-Kernel-NPU Official Repository](https://github.com/sgl-project/sgl-kernel-npu) * [SGL-Kernel-NPU Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md) * [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html) * [PyTorch Custom Ops Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func) * [helloworld Example Operator](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld) * [SGLang Contribution Guide](/docs/hardware-platforms/ascend-npus/development/contribution_guide) # Operator Performance Optimization Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing ## Performance\_benchmark ### Obtaining Performance Data Before optimizing the performance, you need to obtain accurate performance data, understand the current performance status, and analyze the next optimization direction based on the performance status. MindStudio provides realistic methods for testing the performance of Triton operators. #### Device-end The msProf tool is used to collect and analyze key performance indicators of operators running on the Ascend AI Processor. You can use the output performance data to quickly locate the software and hardware performance bottlenecks of operators and improve the efficiency of operator performance analysis. ```bash theme={null} msprof op python3 test_xxxxx.py ``` The following is a case of using msprof for data collection. | Attribute | Value | | ----------------------- | ----------------------------------------------------------------- | | Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 | | Type | DequantSwigluQuant | | OP State | static | | Accelerator Core | AI\_VECTOR\_CORE | | Start Time(us) | 1774489226717521.715 | | Duration(us) | 102.824 | | Wait Time(us) | 0 | | Block Dim | 36 | | Mix Block Dim | 0 | | HF32 Eligible | NO | | Input Shapes | 163840,1024;128,1024;163840;;;;128 | | Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 | | Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | | Output Shapes | 163840,512;163840 | | Output Data Types | INT8;FLOAT | | Output Formats | ND;ND | | Context ID | N/A | | aicore\_time(us) | 0 | | aic\_total\_cycles | 0 | | aic\_mac\_time(us) | 0 | | aic\_mac\_ratio | 0 | | aic\_scalar\_time(us) | 0 | | aic\_scalar\_ratio | 0 | | aic\_mte1\_time(us) | 0 | | aic\_mte1\_ratio | 0 | | aic\_mte2\_time(us) | 0 | | aic\_mte2\_ratio | 0 | | aic\_fixpipe\_time(us) | 0 | | aic\_fixpipe\_ratio | 0 | | aic\_icache\_miss\_rate | 0 | | aiv\_time(us) | 59.128 | | aiv\_total\_cycles | 3512188 | | aiv\_vec\_time(us) | 36.708 | | aiv\_vec\_ratio | 0.621 | | aiv\_scalar\_time(us) | 41.403 | | aiv\_scalar\_ratio | 0.7 | | aiv\_mte2\_time(us) | 11.975 | | aiv\_mte2\_ratio | 0.203 | | aiv\_mte3\_time(us) | 9.738 | | aiv\_mte3\_ratio | 0.165 | | aiv\_icache\_miss\_rate | 0.005 | | cube\_utilization(%) | 0 | Below is the field-by-field breakdown of the operator performance record, aligned with the official specification. ### 1. Basic Identification Fields | Field | Value | Definition (per official docs) | | ---------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. | | Type | DequantSwigluQuant | **OP Type**: Functional category of the operator. | | OP State | static | **OP State**: Indicates a static operator whose shape and scheduling logic are determined at compile time. | | Accelerator Core | AI\_VECTOR\_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI\_CORE (matrix computation core) and AI\_CPU. | ### 2. Timing & Scheduling Fields | Field | Value | Definition (per official docs) | | -------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. | | Duration(us) | 102.824 | **Task Duration**: End-to-end total latency of the operator, including dispatch time, accelerator execution time, and completion response time, in microseconds. | | Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. | ### 3. Core Configuration & Precision Fields | Field | Value | Definition (per official docs) | | ------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. | | Mix Block Dim | 0 | **Mix Block Num**: Number of blocks on the secondary accelerator if the operator runs on both AI Core and Vector Core. A value of 0 means the operator runs exclusively on AI\_VECTOR\_CORE with no hybrid core scheduling. | | HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. | ### 4. Input & Output Information | Field | Value | Definition (per official docs) | | ----------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Input Shapes | 163840,1024;128,1024;16384;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | | Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. | | Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. | | Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. | | Output Data Types | INT8;FLOAT | **Output Data Types**: Output 1 is INT8 (quantized result), output 2 is FLOAT. | | Output Formats | ND;ND | **Output Formats**: Both outputs use standard ND layout. | | Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. | ### 5. AI Core Performance Metrics (aic\_\* series) | Field | Value | Definition (per official docs) | | -------------------------------------------- | ----- | --------------------------------------------------------------------- | | aicore\_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. | | aic\_total\_cycles | 0 | Total execution cycles on AI Core. | | aic\_mac\_time(us) / aic\_mac\_ratio | 0 / 0 | Latency and cycle ratio of cube (matrix multiplication) instructions. | | aic\_scalar\_time(us) / aic\_scalar\_ratio | 0 / 0 | Latency and cycle ratio of scalar instructions. | | aic\_mte1\_time(us) / aic\_mte1\_ratio | 0 / 0 | Latency and cycle ratio of L1→L0A/L0B data move instructions. | | aic\_mte2\_time(us) / aic\_mte2\_ratio | 0 / 0 | Latency and cycle ratio of DDR→AICORE read move instructions. | | aic\_fixpipe\_time(us) / aic\_fixpipe\_ratio | 0 / 0 | Latency and cycle ratio of L0C→OUT/L1 move instructions. | | aic\_icache\_miss\_rate | 0 | Instruction cache miss rate of AI Core. | ### 6. AI Vector Core Performance Metrics (aiv\_\* series) | Field | Value | Definition & Interpretation | | ------------------------------------------ | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aiv\_time(us) | 59.128 | **aiv\_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. | | aiv\_total\_cycles | 3512188 | **aiv\_total\_cycles**: Total cycles executed on Vector Core, summed across all blocks. | | aiv\_vec\_time(us) / aiv\_vec\_ratio | 36.708 / 0.621 | Latency (us) and cycle ratio (62.1%) of vector computation instructions. Vector operations are the core compute workload of this operator. | | aiv\_scalar\_time(us) / aiv\_scalar\_ratio | 41.403 / 0.7 | Latency (us) and cycle ratio (70%) of scalar instructions. The sum exceeds 100% because scalar and vector pipelines run in parallel with independent cycle counters. | | aiv\_mte2\_time(us) / aiv\_mte2\_ratio | 11.975 / 0.203 | Latency and cycle ratio (20.3%) of read memory move instructions (DDR/on-chip memory → Vector Core). | | aiv\_mte3\_time(us) / aiv\_mte3\_ratio | 9.738 / 0.165 | Latency and cycle ratio (16.5%) of write memory move instructions (Vector Core → DDR/on-chip memory). | | aiv\_icache\_miss\_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. | ### 7. Utilization Metrics | Field | Value | Definition (per official docs) | | -------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------- | | cube\_utilization(%) | 0 | **cube\_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. | ## Optimization ### Specification #### 1. Ascend core compute units * AI Core: the core that actually performs matrix/vector computation * Vector Unit: responsible for SIMD computation (similar to CUDA Core) * Scalar Unit: responsible for control/loop * L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64KB, L1 is 256KB, and L2 is shared. #### 2. Ascend memory hierarchy (from fastest to slowest) * Register → Fastest * L0/L1 cache → Very fast * On-chip cache (L2) → Fast * DDR (host memory) → Slowest ### 3. Characteristics of Ascend instructions Good at accessing large contiguous memory blocks Dislikes discrete access, stride access, and random access Must be 128-bit/256-bit aligned Must be vectorized. ### Tips 1. Ascend 910 series usually has only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores. 2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy. 3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode. 4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. 5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1. 6. If hostbound behavior is severe, core binding can be used to address it. # How to Support New Models Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/support_new_models This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations. ## How to Support a New Language Model To support a new model in SGLang, you only need to add a single file under the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn from existing model implementations and create a new file for your model. For most models, you should be able to find a similar model to start with (e.g., starting from Llama). Also refer to how to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang) NPU adaptations are embedded in existing model files (e.g., `llama.py`, `qwen3_vl.py`) through `_is_npu` conditional branches. The NPU hardware backend lives at `sglang/srt/hardware_backend/npu/`. Some ops may need to use `torch_npu` APIs in place of CUDA equivalents. ## How to Support a New Multimodal Large Language Model To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the standard LLM support: 1. **Register your new model as multimodal**: Extend `is_multimodal_model` in [model\_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561) to return `True` for your model. 2. **Register a new chat-template**: Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function. 3. **Multimodal Data Processor**: Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your model’s dedicated processor. See [Multimodal Processors](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors) for more details. 4. **Handle Multimodal Tokens**: Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data with `RadixAttention`. 5. **Handle Image Feature Extraction**: Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model. 6. **Adapt to Vision Attention**: Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`. You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs. On Ascend NPU, ensure vision processors and image feature extraction are compatible with the `torch_npu` backend. Refer to `vit_npu_graph_runner.py` under `hardware_backend/npu/graph_runner/` and `qwen_vl_processor.py` under `hardware_backend/npu/modules/` for NPU vision adaptation patterns. ## Testing and Debugging Please note all your testing and benchmarking results in PR description. ### Benchmark * **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow the [benchmark and profiling guide](/docs/developer_guide/benchmark_and_profiling) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). * **(Optional) Other evals**: If you ran other evals, please note the results in PR description. For NPU-adapted models: add the corresponding test under `test/registered/npu/` and verify correctness on Ascend NPU hardware; run benchmarks on the NPU device and report performance metrics (TTFT, throughput), comparing against SGLang GPU results as the primary baseline. Fall back to HF Transformer comparison when no GPU adaptation is available. ## Port a Model from vLLM to SGLang The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models from vLLM to SGLang. To port a model from vLLM to SGLang: * Compare these two files for guidance: * [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py) * [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py) * The major differences include: * **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`). * **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.** * **(For multimodal models) Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.** * **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers. * **Change the `forward()` functions** to accept a `forward_batch: ForwardBatch` argument carrying the full batch state (positions, `input_ids`, KV cache indices, sampling info, etc.). SGLang’s top-level model `forward()` is the entry point and returns the final `LogitsProcessorOutput` directly — it does both the backbone forward and the logits computation, unlike vLLM which splits these into `forward()` + `compute_logits()`. * **Add `EntryClass`** at the end. * **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components. * **For Ascend NPU**: Reference existing NPU-adapted models (e.g., `llama.py`, `deepseek_v2.py`) for NPU-specific patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at `sglang/srt/hardware_backend/npu/`. Note: make sure you add your new model to the supported models list in the [supported models documentation](https://github.com/sgl-project/sglang/blob/main/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx). ## Registering an External Model Implementation In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server. This allows you to integrate your model without modifying the source code. For example: ```python Register Model theme={null} from sglang.srt.models.registry import ModelRegistry from sglang.srt.entrypoints.http_server import launch_server # For a single model, add it to the registry: ModelRegistry.models[model_name] = model_class # For multiple models, you can imitate the import_model_classes() function: from functools import lru_cache @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {} # Populate model_arch_name_to_cls with your new model classes. ... return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) # Launch the server with your server arguments: launch_server(server_args) ``` ## Example: Implementing and Serving a Llama Wrapper Model Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](/docs/basic_usage/offline_engine_api). ### Implementing Our Model To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit. Let's start by defining our model in a file called `llama_wrapper.py`. The first step is to import the necessary libraries from SRT, which is SGLang's internal backend. ```python Example theme={null} # In the file `llama_wrapper.py` import torch from transformers import LlamaConfig from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM ``` Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`. Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219). Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us. ```python Class Definition theme={null} class LlamaWrapper(LlamaForCausalLM): def __init__( self, config: LlamaConfig, quant_config: Optional[QuantizationConfig] = None, prefix: str = "", ) -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) ``` Now, we want to define the `forward` method, which is what will be called at inference time. Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references. To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py). ```python Forward Method Signature theme={null} @torch.no_grad() def forward( self, input_ids: torch.Tensor, positions: torch.Tensor, forward_batch: ForwardBatch, pp_proxy_tensors: Optional[PPProxyTensors] = None, input_embeds: Optional[torch.Tensor] = None, get_embedding: bool = False, ) -> LogitsProcessorOutput: ``` We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method. After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`). ```python Call Model and LogitsProcessor theme={null} hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) ``` After receiving the logits for the next token, we can finally perform our biasing step. ```python Logit Biasing theme={null} orig_logits = res.next_token_logits res.next_token_logits = torch.where( orig_logits > 0, orig_logits.sqrt(), orig_logits ) return res ``` Now, our `LlamaWrapper` model is created and ready to be served! ### Serving Our Model Via SGLang's Offline Engine The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server. First, create a new file called `run.py`. Now, we must ensure that SGLang's `ModelRegistry` can find our model. To do this, we first download the model's configuration and weights from Huggingface. ```python Example theme={null} # In the file `run.py` import asyncio from functools import lru_cache from huggingface_hub import snapshot_download from llama_wrapper import LlamaWrapper # Make sure to import our new model! import sglang as sgl from sglang.srt.models.registry import ModelRegistry # Make sure to request access to this model on Huggingface, then export your # `HF_TOKEN` to download the model snapshot llama_dir = snapshot_download( repo_id="meta-llama/Llama-3.1-8B-Instruct", local_dir="./llama_ckpt", ) ``` Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`. That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model. ```json Example theme={null} { "architectures": [ "LlamaWrapper" ], ... } ``` However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model. Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation". ```python Register LlamaWrapper theme={null} @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper} return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) ``` Lastly, when we create our `Engine`, we just pass in the path to the local model directory. Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint. ```python Example theme={null} def main(): llm = sgl.Engine(model_path="./llama_ckpt") sampling_params = {"temperature": 0.2, "top_k": 5} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] asyncio.run(run_llm(llm, sampling_params, prompts)) llm.shutdown() async def run_llm( llm, sampling_params, prompts, ) -> None: outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"\nPrompt: {prompt}") print(f"Generated text: {output['text']}") if __name__ == "__main__": main() ``` Now, when we call `python run.py`, we will get the outputs of our newly created model! ## Serving External Models via the Standard CLI The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable. On Ascend NPU, `--device` and `--attention-backend` are auto-detected and can be omitted from the launch command. SGLang sets the device to `npu` and attention backend to `ascend` automatically when `torch.npu.is_available()`. ### The `EntryClass` Variable When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`. For example, if you are implementing a Llama wrapper, add this line at the end of your model file: ```python Example theme={null} # This is what "Add EntryClass at the end" means EntryClass = LlamaWrapper ``` ### Example: Text-Only Model Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI. 1. Create your project ```text theme={null} sglang_custom_project/ |----setup.py |----custom_llm/ |----__init__.py |----llama_wrapper.py ``` Write the `setup.py`: ```python Example theme={null} # sglang_custom_project/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins", version="0.1", packages=find_packages(), ) ``` 2. Write your model code Inside `llama_wrapper.py`, write your model and include `EntryClass`: ```python Example theme={null} # sglang_custom_project/custom_llm/llama_wrapper.py import torch from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM class LlamaWrapper(LlamaForCausalLM): def __init__(self, config, quant_config: Optional[QuantizationConfig] = None, prefix: str = "") -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) @torch.no_grad() def forward(self, input_ids, positions, forward_batch, pp_proxy_tensors=None, input_embeds=None, get_embedding=False): hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res # Don't forget to add EntryClass EntryClass = LlamaWrapper ``` 3. Install your package Run this inside your `sglang_custom_project` directory to install your code into the active Python environment: ```bash Command theme={null} pip install -e . ``` 4. Update your `config.json` Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name: ```json Config theme={null} { "architectures": ["LlamaWrapper"], ... } ``` 5. Launch the server Set the environment variable before running the CLI: ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm python -m sglang.launch_server \ --model-path /path/to/Llama-3.1-8B-Instruct \ --port 8000 ``` The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`. ### Example: Multimodal Model If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor. You can handle this by setting two additional environment variables: * `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models. * `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class. For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits. Create the project: ```text theme={null} sglang_custom_project_vl/ |----setup.py |----custom_vlm/ |----__init__.py |----qwenvl_wrapper.py ``` Write `setup.py`: ```python Example theme={null} # sglang_custom_project_vl/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins-vl", version="0.1", packages=find_packages(), ) ``` Write the model in `qwenvl_wrapper.py`: ```python Example theme={null} # sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py import torch from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor class CustomQwen2VL(Qwen2VLForConditionalGeneration): def forward(self, input_ids, positions, forward_batch, input_embeds=None, get_embedding=False): res = super().forward( input_ids, positions, forward_batch, input_embeds=input_embeds, get_embedding=get_embedding ) if not get_embedding: orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res class CustomQwen2VLProcessor(QwenVLImageProcessor): models = [CustomQwen2VL] def __init__(self, hf_config, server_args, _processor, *args, **kwargs): super().__init__(hf_config, server_args, _processor, *args, **kwargs) EntryClass = CustomQwen2VL ``` **Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class. Install the package, update `config.json`, and launch: ```bash Command theme={null} pip install -e . ``` ```json Config theme={null} { "architectures": ["CustomQwen2VL"], ... } ``` ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm python -m sglang.launch_server \ --model-path /path/to/Qwen2-VL-2B-Instruct \ --port 8000 \ --enable-multimodal ``` ## Documentation Add to table of supported models in [generative\_models.mdx](/docs/supported-models/generative_models) or [multimodal\_language\_models.mdx](/docs/supported-models/multimodal_language_models) For NPU-adapted models, also add entries to the NPU support models table in [reference/support\_models.mdx](../reference/support_models). *** By following these guidelines, you can add support for new language models and multimodal large language models in SGLang and ensure they are thoroughly tested and easily integrated into the system. # Accuracy Evaluation Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation # Ascend NPU Accuracy Evaluation This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered: * **Online Testing**: Evaluate via API interface after starting SGLang server * **Text Models**: Using Qwen2.5-7B-Instruct as example * **Multimodal Models**: Using Qwen2.5-VL-7B-Instruct as example *** ## Environment Setup Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. First, launch the SGLang environment using the provided container image: ```shell Command theme={null} export IMAGE=quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ```shell Command theme={null} export IMAGE=quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` *** ## Using EvalScope [EvalScope](https://github.com/modelscope/evalscope) is a comprehensive model evaluation framework from ModelScope, supporting both accuracy evaluation and performance stress testing. ### Install EvalScope ```shell Command theme={null} # Method 1: Installing via pip pip install evalscope # Method 2: Installing from source git clone https://github.com/modelscope/evalscope.git cd evalscope/ pip install -e . ``` ### Online Text Model Testing This section covers online evaluation scenarios where the SGLang server is already running. #### Start SGLang Server ```shell Command theme={null} # Set HuggingFace mirror (if network access is restricted) export HF_ENDPOINT=https://hf-mirror.com # Start text model server sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 & ``` For more details of SGLang server, refer to the [Ascend NPU Quick Start](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) #### Execute Accuracy Evaluation EvalScope connects to the SGLang server via OpenAI-compatible API. The following example uses the GSM8K dataset: ```shell Command theme={null} evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets gsm8k \ --limit 10 ``` Upon completion, results similar to the following will be displayed: ```text theme={null} +---------------------+-----------+----------+----------+-------+---------+---------+ | Model | Dataset | Metric | Subset | Num | Score | Cat.0 | +=====================+===========+==========+==========+=======+=========+=========+ | Qwen2.5-7B-Instruct | gsm8k | mean_acc | main | 5 | 1.0 | default | +---------------------+-----------+----------+----------+-------+---------+---------+ ``` > **Note**: Output format may vary slightly across different EvalScope versions. The above example is from EvalScope 1.6.x. Ensure the `--model` parameter matches the model name returned by the SGLang server's `/v1/models` endpoint. When starting the server with an HF path (e.g., `Qwen/Qwen2.5-7B-Instruct`), use that path directly. For local paths, pass the full path or the model name returned by `/v1/models`. #### Common Datasets for Online Evaluation ```shell Command theme={null} # MMLU evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mmlu # CEval (Chinese evaluation) evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets ceval # MATH-500 evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets math_500 # HumanEval (code generation) evalscope eval \ --model /home/weights/Qwen2.5-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets humaneval ``` ### Online Multimodal Model Testing #### Start Multimodal Model Server ```shell Command theme={null} # Start multimodal model server (Qwen2.5-VL-7B-Instruct) # Multimodal models require both --attention-backend and --mm-attention-backend sglang serve --model-path /home/weights/Qwen2.5-VL-7B-Instruct \ --attention-backend ascend \ --mm-attention-backend ascend_attn \ --host 0.0.0.0 --port 30000 & ``` #### Execute Multimodal Accuracy Evaluation ```shell Command theme={null} # MMBench (multimodal evaluation) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mm_bench # MMMU (multimodal comprehensive understanding) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets mmmu # HallusionBench (hallucination evaluation) evalscope eval \ --model /home/weights/Qwen2.5-VL-7B-Instruct \ --api-url http://localhost:30000/v1 \ --api-key EMPTY \ --eval-type openai_api \ --datasets hallusion_bench ``` For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/). *** ## Troubleshooting ### SGLang Server Startup Failure 1. Verify device mapping: A2 uses `davinci[0-7]`, A3 uses `davinci[0-15]` 2. Confirm image tag matches device type: A2 uses `...-910b`, A3 uses `...-a3` 3. Check NPU status with `npu-smi info` 4. First run requires model download; set `HF_ENDPOINT=https://hf-mirror.com` if network access is restricted ### EvalScope Connection Failure to Server 1. Confirm SGLang server started successfully (look for `Application startup complete` in logs) 2. Verify `--api-url` points to the correct port (SGLang defaults to `30000`) 3. Ensure URL ends with `/v1`, e.g., `http://localhost:30000/v1` ### EvalScope SSL certificate verification failed When using EvalScope commands without specifying a dataset or model path, it will attempt to download automatically, which may encounter an SSL certificate verification error: ```text theme={null} File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 605, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 592, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 706, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/adapters.py", line 676, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/gsm8k (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)'))) [ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception ``` **Temporary workaround (test only):** Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`. This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production. **Stable solution:** The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle: ```shell theme={null} # Obtain the CA certificate from your network administrator # Then set the environment variable: export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt ``` This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations. If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below. ### EvalScope Request Retry Timeout If EvalScope keeps retrying requests with errors like: ``` 2026-06-22 03:09:03 - evalscope - WARNING: Attempt 4 / 5 failed: ....... Retrying... 2026-06-22 03:09:14 - evalscope - INFO: Evaluating[ceval] 0%| 0/520 [Elapsed: 02:00 < Remaining: ?, ?it/s] 2026-06-22 03:09:19,557 - openai._base_client - INFO: Retrying request to /chat/completions in 0.447260 seconds 2026-06-22 03:09:26,088 - openai._base_client - INFO: Retrying request to /chat/completions in 0.992551 seconds ``` This is usually caused by the HTTP proxy intercepting requests to the local SGLang server. Disable the proxy with: ```shell Command theme={null} unset http_proxy unset https_proxy unset HTTP_PROXY unset HTTPS_PROXY ``` ### Download Dataset Error For this error ```text theme={null} root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --2026-05-12 12:08:01-- https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv Connecting to :... connected. ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘’: Self-signed certificate encountered. To connect to www.modelscope.cn insecurely, use `--no-check-certificate`. ``` You can add `--no-check-certificate` ```bash theme={null} wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate ``` For additional assistance, refer to [SGLang GitHub Issues](https://github.com/sgl-project/sglang/issues). # Performance Testing Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/performance_testing This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools. The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Atlas 800I A2 vs A3), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data. ## 1. Prepare ### 1.1 Start SGLang server Launch the server with the appropriate flags for each model type. Make sure SGLang is installed first — see [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) for environment setup. ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen2.5-7B-Instruct ``` ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct --mm-attention-backend ascend_attn ``` ```shell Command theme={null} # The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded. sglang serve --model-path Qwen/Qwen3-Embedding-8B --is-embedding ``` Add `&` at the end of the command to run the server in the background, or open a new terminal to run the benchmark commands in the following sections. The server binds to `http://127.0.0.1:30000` by default. All online benchmarks below assume the server is running at that address. The `--is-embedding` flag is required for embedding models. ### 1.2 Install benchmarking tools `bench_serving` and `bench_offline_throughput` are built into SGLang and require no extra installation. For Evalscope and AISBench, set up each in its own virtual environment: ```shell Command theme={null} python3 -m venv .evalscope_venv source .evalscope_venv/bin/activate pip install evalscope[perf] -U ``` ```shell Command theme={null} python3 -m venv .aisbench_venv source .aisbench_venv/bin/activate git clone https://github.com/AISBench/benchmark.git cd benchmark/ pip3 install -e ./ --use-pep517 pip3 install -r requirements/api.txt pip3 install -r requirements/extra.txt ``` Run `ais_bench -h` to verify. AISBench requires Python 3.10-3.12. After installation, all AISBench commands must be run from the `benchmark/` directory (the cloned repo root). Set `stream=True` and `ignore_eos=True` in the model config for accurate results. ## 2. Online Service: Text Generation Model Test `Qwen/Qwen2.5-7B-Instruct` via the online serving endpoint. Before running any benchmark in this section, make sure the SGLang text-generation server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) for the launch command. For performance testing, prefer random datasets (`--dataset random`, `--dataset-name random`) over real datasets. Random datasets let you pin `--min-prompt-length` / `--max-prompt-length` and `--min-tokens` / `--max-tokens` to fixed values, producing consistent, repeatable results. Real datasets (ShareGPT, openqa, etc.) have variable input lengths that add noise and make cross-run comparisons unreliable. ### 2.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang server running at `http://127.0.0.1:30000`. Run the following command to run a performance test against the server: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen2.5-7B-Instruct \ --url http://127.0.0.1:30000/v1/chat/completions \ --api openai \ --dataset random \ --max-tokens 1024 \ --min-tokens 1024 \ --prefix-length 0 \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --tokenizer-path Qwen/Qwen2.5-7B-Instruct \ --extra-args '{"ignore_eos": true}' ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} Benchmarking summary: ┌────────────────────────────┬─────────────┐ │ Metric │ Value │ ├────────────────────────────┼─────────────┤ │ ── General ── │ │ │ Test Duration (s) │ 89.34 │ │ Concurrency │ 10 │ │ Request Rate (req/s) │ -1.00 │ │ Total / Success / Failed │ 20 / 20 / 0 │ │ Req Throughput (req/s) │ 0.22 │ │ ── Latency ── │ │ │ Avg Latency (s) │ 44.67 │ │ TTFT (ms) │ 578.51 │ │ TPOT (ms) │ 43.10 │ │ ITL (ms) │ 43.12 │ │ ── Tokens ── │ │ │ Avg Input Tokens │ 1024.00 │ │ Avg Output Tokens │ 1024.00 │ │ Output Throughput (tok/s) │ 229.24 │ │ Total Throughput (tok/s) │ 458.49 │ │ ── Speculative Decoding ── │ │ │ Decoded Tok/Iter │ 1.00 │ │ Spec. Accept Rate │ 0.00 │ └────────────────────────────┴─────────────┘ Percentile results: ┌────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐ │ Metric │ 1% │ 5% │ 10% │ 25% │ 50% │ 75% │ 90% │ 95% │ 99% │ ├────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤ │ Latency (s) │ 44.47 │ 44.47 │ 44.47 │ 44.47 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │ │ TTFT (ms) │ 138.12 │ 142.07 │ 426.17 │ 426.87 │ 783.67 │ 785.26 │ 786.85 │ 787.97 │ 787.97 │ │ ITL (ms) │ 41.84 │ 42.14 │ 42.22 │ 42.36 │ 42.57 │ 42.80 │ 42.99 │ 49.24 │ 49.84 │ │ TPOT (ms) │ 42.71 │ 42.71 │ 42.71 │ 43.05 │ 43.08 │ 43.43 │ 43.43 │ 43.71 │ 43.71 │ │ Input tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ │ Output tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ │ Output (tok/s) │ 22.83 │ 22.83 │ 22.83 │ 22.83 │ 23.02 │ 23.03 │ 23.03 │ 23.03 │ 23.03 │ │ Total (tok/s) │ 45.65 │ 45.65 │ 45.65 │ 45.65 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │ │ Decode (tok/s) │ 22.88 │ 23.03 │ 23.03 │ 23.07 │ 23.21 │ 23.42 │ 23.42 │ 23.42 │ 23.42 │ └────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘ ... ``` See the [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) for full details. ### 2.2 Using AISBench Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands must be run from the `benchmark/` directory. SGLang server running at `http://127.0.0.1:30000`. Set `stream=True` and `ignore_eos=True` in the model config for accurate results. Two files need to be configured for performance testing. First, describe the model and server settings in `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py`: ```python vllm_api_stream_chat.py theme={null} # more details: https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/scenes_intro/performance_benchmark.html from ais_bench.benchmark.models import VLLMCustomAPIChat from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content models = [ dict( attr="service", type=VLLMCustomAPIChat, abbr="vllm-api-stream-chat", path="Qwen/Qwen2.5-7B-Instruct", model="Qwen/Qwen2.5-7B-Instruct", stream=True, request_rate=0, use_timestamp=False, retry=2, api_key="", host_ip="127.0.0.1", host_port=30000, url="", max_out_len=512, batch_size=32, trust_remote_code=False, generation_kwargs=dict( temperature=0.01, ignore_eos=True, ), pred_postprocessor=dict(type=extract_non_reasoning_content), ) ] ``` If the model has already been downloaded, point `path` to the local model path instead of the model id. Second, configure random prompt lengths in `ais_bench/datasets/synthetic/synthetic_config.py`: ```python synthetic_config.py theme={null} # more details: https://ais-bench-benchmark.readthedocs.io/en/latest/advanced_tutorials/synthetic_dataset.html synthetic_config = { "Type":"tokenid", "RequestCount": 10, "TrustRemoteCode": False, "StringConfig" : { "Input" : { "Method": "uniform", "Params": {"MinValue": 1, "MaxValue": 200} }, "Output" : { "Method": "gaussian", "Params": {"Mean": 100, "Var": 200, "MinValue": 1, "MaxValue": 100} } }, "TokenIdConfig" : { "RequestSize": 10, "PrefixLen": 0 } } ``` Run with a synthetic dataset: ```shell Command theme={null} ais_bench --models vllm_api_stream_chat --datasets synthetic_gen_string -m perf ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ╒══════════════════════════╤═════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════╕ │ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │ ╞══════════════════════════╪═════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════╡ │ E2EL │ total │ 3896.4 ms │ 3081.6 ms │ 4175.3 ms │ 4013.8 ms │ 4123.4 ms │ 4137.1 ms │ 4171.5 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ TTFT │ total │ 411.6 ms │ 346.7 ms │ 439.7 ms │ 416.3 ms │ 426.6 ms │ 434.4 ms │ 439.2 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ TPOT │ total │ 38.3 ms │ 37.4 ms │ 39.0 ms │ 38.3 ms │ 38.7 ms │ 38.9 ms │ 39.0 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ ITL │ total │ 38.7 ms │ 0.0 ms │ 156.5 ms │ 38.9 ms │ 39.0 ms │ 39.2 ms │ 117.1 ms │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ InputTokens │ total │ 123.4 │ 34.0 │ 228.0 │ 130.5 │ 170.5 │ 217.2 │ 226.92 │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ OutputTokens │ total │ 92.1 │ 69.0 │ 100.0 │ 95.0 │ 99.75 │ 100.0 │ 100.0 │ 10 │ ├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤ │ OutputTokenThroughput │ total │ 23.5937 token/s │ 22.3912 token/s │ 24.2616 token/s │ 23.7399 token/s │ 23.9919 token/s │ 24.2027 token/s │ 24.2557 token/s │ 10 │ ╘══════════════════════════╧═════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════╛ ╒══════════════════════════╤═════════╤══════════════════╕ │ Common Metric │ Stage │ Value │ ╞══════════════════════════╪═════════╪══════════════════╡ │ Benchmark Duration │ total │ 4175.4485 ms │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Requests │ total │ 10 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Failed Requests │ total │ 0 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Success Requests │ total │ 10 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Concurrency │ total │ 9.3317 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Max Concurrency │ total │ 32 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Request Throughput │ total │ 2.395 req/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Input Tokens │ total │ 1234 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Prefill Token Throughput │ total │ 299.8329 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Generated Tokens │ total │ 921 │ ├──────────────────────────┼─────────┼──────────────────┤ │ Input Token Throughput │ total │ 295.5371 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Output Token Throughput │ total │ 220.5751 token/s │ ├──────────────────────────┼─────────┼──────────────────┤ │ Total Token Throughput │ total │ 516.1122 token/s │ ╘══════════════════════════╧═════════╧══════════════════╛ ``` See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details. ### 2.3 Using bench\_serving SGLang's built-in `bench_serving` requires no extra installation. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark. See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for all backends, datasets, and advanced options. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang-oai \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 512 \ --random-range-ratio 1 \ --num-prompts 100 \ --max-concurrency 32 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically. 1. If you have network issues, set `export HF_ENDPOINT=https://hf-mirror.com` to use domestic mirror. 2. If downloading still fails, manually download the dataset file `ShareGPT_V3_unfiltered_cleaned_split.json` locally, upload it to your server, then specify the file directory via `--dataset-path` to run offline. Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving). Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang-oai Traffic request rate: inf Max request concurrency: 32 Successful requests: 100 Benchmark duration (s): 47.51 Total input tokens: 102400 Total input text tokens: 102400 Total generated tokens: 51200 Total generated tokens (retokenized): 51195 Request throughput (req/s): 2.10 Input token throughput (tok/s): 2155.35 Output token throughput (tok/s): 1077.68 Peak output token throughput (tok/s): 1587.00 Peak concurrent requests: 64 Total token throughput (tok/s): 3233.03 Concurrency: 26.93 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 12793.49 Median E2E Latency (ms): 12940.17 P90 E2E Latency (ms): 13049.86 P99 E2E Latency (ms): 13051.61 ---------------Time to First Token---------------- Mean TTFT (ms): 1423.99 Median TTFT (ms): 1489.29 P99 TTFT (ms): 2325.56 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 22.25 Median TPOT (ms): 22.22 P99 TPOT (ms): 25.08 ---------------Inter-Token Latency---------------- Mean ITL (ms): 22.26 Median ITL (ms): 20.74 P95 ITL (ms): 21.40 P99 ITL (ms): 23.62 Max ITL (ms): 2229.30 ================================================== ``` #### SGLang Serving Benchmark Result — Complete Reference The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments. ##### Test Configuration
Parameter Description
Backend The serving backend under test (e.g., sglang, vllm).
Traffic request rate Request generation rate in req/s. inf means maximum rate (concurrency-bounded). trace indicates trace timestamp mode. A fixed value enforces constant inter-arrival time.
Max request concurrency Maximum number of concurrent requests from the client side. Displays not set when unspecified.
##### Core Statistics & Throughput Metrics
Parameter Description Format Specification
Successful requestsTotal number of successfully completed requests (HTTP 200, no generation errors).Integer, no decimal places
Benchmark duration (s)Total elapsed time from first request sent to last response fully received (seconds).2 decimal places
Total input tokensTotal number of input (prompt) tokens across all requests, counted by server-side tokenizer.Integer, no decimal places
Total input text tokensSame as Total input tokens. For multimodal inputs, this may differ.Integer, no decimal places
Total generated tokensTotal number of output tokens actually generated by the server (server-side tokenizer count).Integer, no decimal places
Total generated tokens (retokenized)Output text re-tokenized by the client using its own tokenizer. A large discrepancy indicates tokenizer mismatch or special tokens in output.Integer, no decimal places
Request throughput (req/s)Number of successful requests processed per second. Formula: Successful requests / Benchmark duration (s).2 decimal places
Input token throughput (tok/s)Number of input tokens processed per second. Formula: Total input tokens / Benchmark duration (s).2 decimal places
Output token throughput (tok/s)Number of output tokens generated per second. Formula: Total generated tokens / Benchmark duration (s).2 decimal places
Peak output token throughput (tok/s)Observed instantaneous peak output token generation rate during the test (computed over a sliding window).2 decimal places
Peak concurrent requestsMaximum number of requests being processed simultaneously on the server side. May exceed client-side Max request concurrency due to queueing.Integer, no decimal places
Total token throughput (tok/s)Sum of input and output token throughputs. Formula: Input token throughput + Output token throughput.2 decimal places
ConcurrencyAverage number of concurrent requests during the test (Little's Law). Formula: Sum of all E2E latencies / Benchmark duration.2 decimal places
##### End-to-End Latency (E2E Latency)
StatisticDescriptionFormat
Mean E2E Latency (ms)Arithmetic mean2 decimal places
Median E2E Latency (ms)50th percentile2 decimal places
P90 E2E Latency (ms)90th percentile (90% of requests have latency ≤ this value)2 decimal places
P99 E2E Latency (ms)99th percentile2 decimal places
##### Time to First Token (TTFT)
StatisticDescriptionFormat
Mean TTFT (ms)Arithmetic mean2 decimal places
Median TTFT (ms)50th percentile2 decimal places
P99 TTFT (ms)99th percentile2 decimal places
##### Time per Output Token (TPOT) – Excluding First Token Formula: (E2E Latency - TTFT) / (Number of output tokens - 1)
StatisticDescriptionFormat
Mean TPOT (ms)Arithmetic mean2 decimal places
Median TPOT (ms)50th percentile2 decimal places
P99 TPOT (ms)99th percentile2 decimal places
##### Inter-Token Latency (ITL)
StatisticDescriptionFormat
Mean ITL (ms)Average inter-token interval2 decimal places
Median ITL (ms)50th percentile inter-token interval2 decimal places
P95 ITL (ms)95th percentile (used to detect stalls)2 decimal places
P99 ITL (ms)99th percentile2 decimal places
Max ITL (ms)Maximum observed inter-token interval; useful for identifying severe blocking events2 decimal places
## 3. Online Service: Multimodal Model Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks. Before running any benchmark in this section, make sure the SGLang multimodal server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Multimodal tab for the launch command. For consistent, repeatable results, set `--random-range-ratio 1` to fix input/output lengths, or `0` (default) for uniform distribution. ### 3.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang multimodal server running at `http://127.0.0.1:30000`. Evalscope's `perf` tool uses the OpenAI-compatible `/v1/chat/completions` endpoint. Use `--dataset random_vl` for randomized multimodal data with image generation: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --url http://127.0.0.1:30000/v1/chat/completions \ --api openai \ --dataset random_vl \ --min-tokens 1024 \ --max-tokens 1024 \ --prefix-length 0 \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --image-width 512 \ --image-height 512 \ --image-format RGB \ --image-num 1 \ --tokenizer-path Qwen/Qwen2.5-VL-7B-Instruct \ --extra-args '{"ignore_eos": true}' ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. ### 3.2 Using AISBench Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands run from the `benchmark/` directory. SGLang multimodal server running at `http://127.0.0.1:30000`. AISBench does not include a built-in multimodal dataset — you must provide your own. First, edit `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py` to configure the vision model: ```python vllm_api_stream_chat.py theme={null} from ais_bench.benchmark.models import VLLMCustomAPIChat from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content models = [ dict( attr="service", type=VLLMCustomAPIChat, abbr="vllm-api-stream-chat", path="Qwen/Qwen2.5-VL-7B-Instruct", model="Qwen/Qwen2.5-VL-7B-Instruct", stream=True, request_rate=0, use_timestamp=False, retry=2, api_key="", host_ip="127.0.0.1", host_port=30000, url="", max_out_len=256, batch_size=16, trust_remote_code=False, generation_kwargs=dict( temperature=0.01, ignore_eos=True, ), pred_postprocessor=dict(type=extract_non_reasoning_content), ) ] ``` If the model has already been downloaded, point `path` to the local model path instead of the model id. Next, download a multimodal dataset such as mmstar: ```shell Command theme={null} # Download the mmstar dataset (from within the benchmark/ directory) cd ais_bench/datasets mkdir mmstar cd mmstar wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv ``` Run the performance test: ```shell Command theme={null} ais_bench --models vllm_api_stream_chat --datasets mmstar_gen -m perf ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ╒══════════════════════════╤═════════╤═════════════════╤════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤══════╕ │ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │ ╞══════════════════════════╪═════════╪═════════════════╪════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪══════╡ │ E2EL │ total │ 6190.9 ms │ 5071.4 ms │ 8464.8 ms │ 6126.6 ms │ 6475.2 ms │ 6833.5 ms │ 7897.9 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ TTFT │ total │ 693.3 ms │ 96.0 ms │ 2161.5 ms │ 747.4 ms │ 870.9 ms │ 1032.3 ms │ 1620.8 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ TPOT │ total │ 21.6 ms │ 17.8 ms │ 32.1 ms │ 21.3 ms │ 23.1 ms │ 24.5 ms │ 29.1 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ ITL │ total │ 25.5 ms │ 0.0 ms │ 1951.1 ms │ 18.8 ms │ 19.7 ms │ 37.3 ms │ 121.8 ms │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ InputTokens │ total │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ OutputTokens │ total │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 1500 │ ├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤ │ OutputTokenThroughput │ total │ 41.6779 token/s │ 30.243 token/s │ 50.4791 token/s │ 41.7847 token/s │ 44.6424 token/s │ 45.6484 token/s │ 46.0932 token/s │ 1500 │ ╘══════════════════════════╧═════════╧═════════════════╧════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧══════╛ ╒═════════════════════════╤═════════╤══════════════════╕ │ Common Metric │ Stage │ Value │ ╞═════════════════════════╪═════════╪══════════════════╡ │ Benchmark Duration │ total │ 582099.6816 ms │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Requests │ total │ 1500 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Failed Requests │ total │ 0 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Success Requests │ total │ 1500 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Concurrency │ total │ 15.9532 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Max Concurrency │ total │ 16 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Request Throughput │ total │ 2.5769 req/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Input Tokens │ total │ 0 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Generated Tokens │ total │ 384000 │ ├─────────────────────────┼─────────┼──────────────────┤ │ Input Token Throughput │ total │ 0.0 token/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Output Token Throughput │ total │ 659.6808 token/s │ ├─────────────────────────┼─────────┼──────────────────┤ │ Total Token Throughput │ total │ 659.6808 token/s │ ╘═════════════════════════╧═════════╧══════════════════╛ ``` See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details. ### 3.3 Using bench\_serving (image dataset) Set `--dataset-name image` for image datasets. `bench_serving` will generate random prompts with image inputs. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark. See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for the full list of image-related flags. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen2.5-VL-7B-Instruct \ --dataset-name image \ --random-input-len 1024 \ --random-output-len 512 \ --random-range-ratio 1 \ --num-prompts 32 \ --max-concurrency 16 \ --image-count 1 \ --image-resolution 720p ``` Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 16 Successful requests: 32 Benchmark duration (s): 51.74 Total input tokens: 73464 Total input text tokens: 35128 Total input vision tokens: 38336 Total generated tokens: 16384 Total generated tokens (retokenized): 9300 Request throughput (req/s): 0.62 Input token throughput (tok/s): 1419.96 Output token throughput (tok/s): 316.68 Peak output token throughput (tok/s): 800.00 Peak concurrent requests: 32 Total token throughput (tok/s): 1736.64 Concurrency: 15.98 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 25841.84 Median E2E Latency (ms): 25842.85 P90 E2E Latency (ms): 26296.42 P99 E2E Latency (ms): 26303.13 ---------------Time to First Token---------------- Mean TTFT (ms): 12211.59 Median TTFT (ms): 14405.77 P99 TTFT (ms): 15837.60 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 26.67 Median TPOT (ms): 21.75 P99 TPOT (ms): 41.89 ---------------Inter-Token Latency---------------- Mean ITL (ms): 26.67 Median ITL (ms): 20.34 P95 ITL (ms): 20.85 P99 ITL (ms): 21.70 Max ITL (ms): 11309.91 ================================================== ``` ## 4. Online Service: Embedding Model Test `Qwen/Qwen3-Embedding-8B` on the embedding API endpoint. Before running any benchmark in this section, make sure the SGLang embedding server is running with `--is-embedding` at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Embedding tab for the launch command. AISBench does not support embedding endpoints — use `bench_serving` or Evalscope instead. ### 4.1 Using Evalscope Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang embedding server running with `--is-embedding` at `http://127.0.0.1:30000`. Evalscope supports embedding evaluation. For performance testing the embedding API directly: ```shell Command theme={null} evalscope perf \ --parallel 10 \ --number 20 \ --model Qwen/Qwen3-Embedding-8B \ --url http://127.0.0.1:30000/v1/embeddings \ --api openai_embedding \ --dataset random_embedding \ --min-prompt-length 1024 \ --max-prompt-length 1024 \ --tokenizer-path Qwen/Qwen3-Embedding-8B ``` If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id. Evalscope's embedding performance testing support may vary by version. If the `perf` command does not accept the embeddings endpoint, use [`bench_serving` with `--backend sglang-embedding`](#4-2-using-bench_serving-embedding-backend) as the primary option. ### 4.2 Using bench\_serving (embedding backend) `bench_serving` is built into SGLang. Use `--backend sglang-embedding` to target the `/v1/embeddings` endpoint. Make sure the server is running with `--is-embedding` at `http://127.0.0.1:30000`. ```shell Command theme={null} python -m sglang.bench_serving \ --backend sglang-embedding \ --base-url http://127.0.0.1:30000 \ --model Qwen/Qwen3-Embedding-8B \ --dataset-name random \ --random-input-len 512 \ --random-output-len 0 \ --num-prompts 1000 \ --max-concurrency 64 \ --request-rate 32 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. Set `--random-output-len 0` for embedding benchmarks — no output tokens are generated. Example output (for illustration only — actual results depend on your hardware and configuration): ```text theme={null} ============ Serving Benchmark Result ============ Backend: sglang-embedding Traffic request rate: 32.0 Max request concurrency: 64 Successful requests: 1000 Benchmark duration (s): 31.86 Total input tokens: 257891 Total input text tokens: 257891 Request throughput (req/s): 31.39 Input token throughput (tok/s): 8094.67 Peak concurrent requests: 62 Concurrency: 6.67 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 212.34 Median E2E Latency (ms): 160.97 P90 E2E Latency (ms): 267.31 P99 E2E Latency (ms): 1445.94 ================================================== ``` ## 5. Offline Performance Testing SGLang's `Engine` API runs inference in-process, without an HTTP server, letting you measure maximum throughput. `bench_offline_throughput` is built into SGLang and requires no extra installation or running server. `bench_offline_throughput` currently only supports text-generation (LLM) benchmarks. Multimodal and embedding models are not supported. ### 5.1 Using bench\_offline\_throughput `bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead: ```shell Command theme={null} python -m sglang.bench_offline_throughput \ --model-path Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 512 \ --num-prompts 500 ``` `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. `--dataset-name random` with `--random-input-len` and `--random-output-len` gives you full control over input/output token counts. Fixed-length random data eliminates variance from real datasets, making throughput comparisons across runs deterministic and reliable. ## See also * [Bench Serving Guide](/docs/developer_guide/bench_serving) — all backends, datasets, and advanced options for `bench_serving` * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — environment setup for Ascend NPUs * [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) — full Evalscope documentation * [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) — full AISBench documentation # Troubleshooting and FAQ Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/faq Contributions are welcome. Feel free to add more. ## 1. Context corruption with GLOO op.preamble.length \<= op.nbytes in PD disaggregation ### Error message ```text highlight=2,5-7 theme={null} [2026-04-07 13:24:13 TP0] Decode batch, #running-req: 10, #token: 485248, token usage: 0.94, pre-allocated usage: 0.51, #prealloc-req: 1, #transfer-req: 12, #retracted-req: 0, npu graph: True, gen throughput (token/s): 259.82, #queue-req: 0 [2026-04-07 13:24:13 TP0] Context corruption detected: Request 3b5dcfe1575d4e1f9b18c953de878a93 (bootstrap_room=7451500070298748792) received metadata from bootstrap_room=4125156593077881415. Metadata buffer index: 1. This indicates metadata buffer index collision. [2026-04-07 13:24:13] INFO: 127.0.0.1:59272 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2026-04-07 13:24:13] INFO: 127.0.0.1:34000 - "POST /v1/chat/completions HTTP/1.1" 200 OK terminate called after throwing an instance of 'gloo::EnforceNotMet' what(): [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:456] op.preamble.length <= op.nbytes. 4 vs 3 Fatal Python error: Aborted Thread 0x0000fff873f6f120 (most recent call first): File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 1499 in heartbeat_checker ``` ### Cause (Possibly, not precisely located) High-concurrency long sequences fill up the transfer buffer, causing buffer index collision and data corruption. ### Solution 1. Disable overlap on the Prefill node with `--disable-overlap-schedule`. The Prefill node in PD disaggregation must not enable overlap, otherwise it causes timing issues that lead to out-of-order reception on the Decode node. 2. Even with overlap disabled, multi-Prefill node high-concurrency long-sequence scenarios may still encounter this issue with low probability. This is a known issue pending resolution. ## 2. Graph mode aclnnInplaceFillScalar error ### Error message ```text highlight=2 theme={null} (SGLangEngine pid=3872176) [rank0]:[E414 12:14:41.204711510 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:26 NPU function error: call aclnnInplaceFillScalar failed, error code is 507000 (SGLangEngine pid=3872176) [ERROR] 2026-04-14-12:14:41 (PID:3874122, Device:0, RankID:-1) ERR00100 PTA call acl api failed (SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host. (SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log. (SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKerneWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog. (SGLangEngine pid=3872176) TraceBack (most recent call last): (SGLangEngine pid=3872176) Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148] (SGLangEngine pid=3872176) aclrtLaunchKerneWWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Launch kernel failed. (SGLangEngine pid=3872176) #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o (SGLangEngine pid=3872176) Kernel Run failed. opType: 18, Fill (SGLangEngine pid=3872176) launch failed for Fill, errno:361001. (SGLangEngine pid=3872176) (SGLangEngine pid=3872176) Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler depend.ts:26 (most recent call first): (SGLangEngine pid=3872176) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >)+ 0xb0 (0xffff806848c0 in /root/anaconda3/envs/slime_re/Lib/python3.11/site-packages/torch/lib/libc10.so) (SGLangEngine pid=3872176) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0x68(0xffff8062c140 in /root/anaconda3/envs/slime_re/Tib/puthon3.11/site-packages/torch/lib/libc10.so) (SGLangEngine pid=3872176) frame #2: + 0x110e2b4 (0xffff6d44e2b4 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #3: + 0x29f0894(0xffff6ed30894 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #4: + 0x9cc708(0xffff6cd0c700 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) (SGLangEngine pid=3872176) frame #5: + 0x9cd2dc (0xffff6cd0d2dc in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so) ``` ### Cause Too many captured graphs cause a conflict in the graph mode update stream. Each graph is placed on a separate stream, but the number of streams is limited. If too many graphs are captured, conflicts occur. ### Solution * CANN 8.5 + TorchNPU 2.8 should have resolved this issue. * If your versions do not match, reduce the number of captured graphs to 10 or fewer. ## 3. alloc\_extend\_kernel error ### Error message ```text theme={null} [root@os-node-created-6z9tp pd_7p1d_tp2_20260414_030537]# grep -nR -E "aivec error|ACL" *,log prefill 1.log:5739:EZ9999[PID: 164243] 2026-04-14-06:40:35.033.745 (EZ9999): The error from device(chipId:0, dieId:1), serial number is 1, there is an exception of aivec error, core id is 23, error code = 0, dump info: pc start: 0x12420156a000, current: 0x12420156ad74, vec error info: 0xdb1751f50e, mte error info: 0x98f6388707, ifu error info: 0x212c93fc00000, ccu error info: 0x998e981c00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc00.[FUNC:PrintCoreInfo][FILE:device error core proc.cc][LINE:347] prefill_1.log:5747:EZ9999[PID: 164242] 2026-04-14-06:40:35.034.456 (EZ9999): The error from device(chipId:0, dieId:0), serial number is 1, there is an exception of aivec error, core id is 45, error code = O, dump info: pc start: 0x12400156a000, current: 0x12400156ad74, vec error info: 0x7304583407, mte error info: 0x27e2b1c765, ifu error info: 0x212c93fc00000, ccu error info: 0x6e0836f300000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc0O.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill_1.Log:5784:RuntimeError: ACL stream synchronize failed, error code:507035 prefill_1.Log:5816:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 2.log:2566:EZ9999[PID: 163475] 2026-04-14-05:28:43.282.016 (EZ9999): The error from device(chipId:1, dieId:1), serial number is 1, there is an exception of aivec error, core id is 20, error code = 0, dump info: pc start: 0x124601550000, current: 0x124601550d74, vec error info: 0x9b07e496e3, mte error info: 0xc2a6806828, ifu error info: 0x212c93f200000, ccu error info: 0xaa64401100000000, cube error info: O, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core proc.cc][LINE:347] prefill 2.log:2574:EZ9999[PID: 163474] 2026-04-14-05:28:43.282.809 (EZ9999): The error from device(chipId:1, dieId:0), serial number is 1, there is an exception of aivec error, core id is 22, error code = 0, dump info: pc start: 0x124401550000, current: 0x124401550d74, vec error info: 0xa80496049a, mte error info: 0xa0770e2bf2, ifu error info: 0x212c93f200000, ccu error info: 0x1083000000000000, cube error info: O, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 2.log:2611:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 2.log:2644:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 4.log:1323:EZ9999[PID: 163478] 2026-04-14-05:08:15.363.509 (EZ9999): The error from device(chipId:3, dieId:0), serial number is 1, there is an exception of aivec error, core id is 5, error code = 0, dump info: pc start: 0x124c00dff000, current: 0x124c00dff860, vec error info: 0xf311fb8727, mte error info: 0x45418486a, ifu error info: 0x212c93fa00000, ccu error info: 0x17a9996f00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 4.log:1331:EZ9999[PID: 163479] 2026-04-14-05:08:15.364.016 (EZ9999): The error from device(chipId:3, dieId:1), serial number is 1, there is an exception of aivec error, core id is 40, error code = 0, dump info: pc start: 0x124e00dff000, current: 0x124e00dff860, vec error info: 0xf11a9cf188, mte error info: 0xfb7710514a, ifu error info: 0x212c93fa00000, ccu error info: 0x1f5828a900000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 4.Log:1368:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 4.log:1400:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 7.log:3017:EZ9999[PID: 164628] 2026-04-14-05:34:12.369.755 (EZ9999): The error from device(chipId:6, dieId:0), serial number is 1, there is an exception of aivec error, core id is 33, error code = 0, dump info: pc start: 0x1258015ae000, current: 0x1258015aed74, vec error info: 0x740053222c, mte error info: 0x97103df5a0, ifu error info: 0x212c93f400000, ccu error info: 0x391c89ab00000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 7.log:3025:EZ9999[PID: 164629] 2026-04-14-05:34:12.369.742 (EZ9999): The error from device(chipId:6, dieId:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = 0, dump info: pc start: 0x125a015ae000, current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] prefill 7.log:3062:RuntimeError: ACL stream synchronize failed, error code:507035 prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:507035 ``` ### Error plog ```text highlight=21 theme={null} [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.353.461 [stars engine.cc:1534]170327 ProcLogicCaReport:Task run failed, device id=13, stream id=43, task id=13145, sqe type=0(ffts), errType=0x1(task exception), sqSwStatus=0 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.720 device error core proc.cc:3211170327 AddExceptionReqInfo:add error register: core id=42, stream id=43, task id=13145 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.730 device error core proc.cc:3471170327 PrintCorelnfo:The error from device(chipld:6, dield:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = O, dump info: pc start: 0x125a015ae000,current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100cd6400. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.774 device error core proc.cc:3601170327 PrintCorelnto:The extend into: errcode:(0, 0x8000, 0) errorStr: When the D-cache reads and writes data to the UB, the response value returned bv the bus is a non-zero value. fixp_error0 info: 0x670ef, fixp error1 info: 0xfc, fsmId:0, tslot:2, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.787 device error core proc.cc:4341170327 ProcessStarsCoreErrorInfo:devId=13, streamId=43, taskId=13145, MTE errorCode=0. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.795 davinci task.cc:2011170327 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.644 davinci kernel task.cc:15821170327 PreCheckTaskErr:Kernel task happen error retCode=0x31, vector core exception1. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.705 davinci kernel task.cc:14241170327 GetArasInfo:[AIC INFO] aras(0 to 9) after execute:0x3fffffb9000, 0, 0, 0x12c9323ff600, 0x12c93231ee00, 0x12c93f1d8800,0x12c93f3ff600,0x12c958200000,0x100000003, 0xaaaa00000001. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.710 [davinci kernel task.cc:1427]170327 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(10*8), argsSize=80, blockDim=3 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.717 [davinci kernel task.cc:1468]170327 PrintErrorInfoForDavinciTask:[AIC INFO] after execute:arqs print end [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend _kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.823 [dump manager.cpp:41][tid:170327] An exception callback message is received. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.971 [kernel info collector.cpp:384][tid:170327] Get error register information. coreNum=0 [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.981 kernel info collector.cpp:4771tid:1703271 It is Non-SuperKernel. functionCount=1, qlobalCount=1 [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.987 [dump args.cpp:668][tid:170327] In arqAddr[0x12c100cd6400]|arqSize[80]dfxAddr[(nil)]|dfxSize[0] has invalid attribute. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.383.894 [dump_printf.cpp:1118][tid:170327] infoAddr is null [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.908 [stream.cc:1332]170327 GetError:Stream Synchronize failed, stream id=43, retCode=0x31, [vector core exception]. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.911 [stream.cc:1335]170327 GetError:AIV Kernel happen error, retCode=0x31. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.929 [stream.cc:1335]170327 GetError:[AIC_INFO] after execute:args print end [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.936 stream.cc:13351170327 GetError: DFX INFO1Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, tlip num=56, fault kernel_name=alloc_extend_kernel_18, ault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.943 [stream.cc:3549]170327 EnterFailureAbort:stream id=43 enter failure abort. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.973 [stars_engine.cc:1427]170327 StarsResumeRtsa:stop scheduling in abort failure mode: stream id=43, sq id=6,sq head=801, task id=13145, taskType=66. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.972 [stream.cc:1463]164629 SynchronizeExecutedTask:context is abort, status=0x715005e. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.978 [stream.cc:1516]164629 Synchronizelmpl:failed, stream_id=43, error=0x715005e [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.982 [api error.cc:1015]164629 StreamSynchronize:Stream synchronize failed, stream_id=43, timeout=-1ms. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.991 [apic stream.cc:154]164629 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.997 [error_message_manage.cc:61]164629 FuncErrorReason:rtStreamSynchronize execution failed, reason=vector core exception [ERROR] ASCENDCL(164629,):2026-04-14-05:34:12.384.269 [stream.cpp:140]164629 acIrtSynchronizeStreamImpl:synchronize stream failed, runtime result = 507035 ``` ### Cause The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pending resolution. ### Solution Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead. ```python highlight=15-36 theme={null} def alloc_extend( self, prefix_lens: torch.Tensor, prefix_lens_cpu: torch.Tensor, seq_lens: torch.Tensor, seq_lens_cpu: torch.Tensor, last_loc: torch.Tensor, extend_num_tokens: int, num_new_pages: int = None, ): ... if num_new_pages_item > len(self.free_pages): return None # if num_new_pages_item < 200: # from sgl_kernel_npu.mem_cache.allocator import alloc_extend_kernel # out_indices = torch.empty( # (extend_num_tokens,), # dtype=torch.int64, # device=self.device, # ) # max_num_extend_tokens = next_power_of_2(extend_num_tokens) # bs = prefix_lens.shape[0] # alloc_extend_kernel[(bs,)]( # prefix_lens, # seq_lens, # last_loc, # self.free_pages, # out_indices, # next_power_of_2(bs), # self.page_size, # max_num_extend_tokens, # ) # else: out_indices = torch.empty( (extend_num_tokens,), dtype=torch.int32, device=self.device, ) ... ``` ## 4. Out of NPU memory ```text highlight=7 theme={null} File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 175, in calculate_pool_sizes return MemoryPoolConfig(max_total_num_tokens=max_total_num_tokens) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 8, in __init__ File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 44, in __post_init__ raise RuntimeError(msg) RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. ``` ### Solution First, use the `npu-smi info` command to check the NPU memory usage. If the NPUs are occupied by other processes, use `--base-gpu-id` to specify the starting device index. If the NPUs are not occupied, you can use `--tp` to deploy across multiple devices, or reduce the KV cache memory usage by increasing the `--mem-fraction-static` value. For detailed tuning guidance, see [Hyperparameter Tuning](/docs/advanced_features/hyperparameter_tuning). ## 5. How to update sgl-kernel-npu ### Solution ```bash theme={null} git clone https://github.com/sgl-project/sgl-kernel-npu.git source /usr/local/Ascend/ascend-toolkit/set_env.sh cd sgl-kernel-npu # Building Project bash build.sh pip install output/sgl_kernel_npu*.whl --force-reinstall # (Optional) Confirm whether the import can be successfully python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)" rm -rf sgl-kernel-npu ``` ## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets ### Error message ```text highlight=1-2 theme={null} '[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json Retrying in 1s [Retry 1/5]. Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in run_benchmark(args) File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark input_requests = get_dataset(args, tokenizer, model_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset return dataset.load(tokenizer=tokenizer, model_id=model_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load return sample_random_requests( ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests dataset_path = download_and_cache_hf_file( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... RuntimeError: Cannot send a request, as the client has been closed. [ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception ``` ### Cause The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations). ### Solution * **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com): ```bash theme={null} export HF_ENDPOINT=https://hf-mirror.com ``` * **Use a proxy** — if you have an HTTP proxy available: ```bash theme={null} export http_proxy=http://your-proxy:port export https_proxy=http://your-proxy:port ``` * **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path: ```bash theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ ... ``` ## 7. Unexpected type fp8 ### Cause FP8 model is not supported ### Solution Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B-w8a8-mtp. ## 8. Docker image versions: stable release vs. daily build Docker images for Ascend NPU are available in two types: * **Stable release** — validated version with a specific tag, e.g., `quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16`. Recommended for production deployments. * **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann8.5.0-a3`. Use this if you need the latest features or bug fixes that have not yet been included in a stable release. If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version. # Installation Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services. You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. ## Component Version Mapping For SGLang
Component Version Obtain Method
HDK 25.5.2 link
CANN 9.0.0 Obtain Images
TorchNPU 26.0.0 link
MemFabric 1.0.8 `pip install memfabric-hybrid==1.0.8`
Triton 3.2.1.dev20260530 `pip install triton-ascend==3.2.1.dev20260530 \`
`--extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \`
`--trusted-host mirrors.huaweicloud.com`
SGLang NPU Kernel 2026.05.01.post3 link
MemFabric-zbal 1.1.1 `pip install memfabric-zbal==1.1.1`
### Obtain CANN Image Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. You can obtain the dependency of a specified version of CANN through an image. ```bash Command theme={null} docker pull quay.io/ascend/cann:9.0.0-a3-ubuntu22.04-py3.11 ``` ```bash Command theme={null} docker pull quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11 ``` ## Preparing the Running Environment ### Method 1: Installing from source with prerequisites #### Python Version **Only `python==3.11` is supported currently**. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda). ```bash Command theme={null} conda create --name sglang_npu python=3.11 conda activate sglang_npu ``` Note on Anaconda repository restrictions If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror): ```bash Command theme={null} # Add Tsinghua mirrors conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/ conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/ conda config --set show_channel_urls yes conda config --remove channels defaults ``` Edit the system-level conda config to remove any hardcoded defaults, e.g., vi \~/miniconda3/.condarc Then remove the failed environment and recreate it: ```bash Command theme={null} conda clean -i conda env remove -n sglang_npu conda create --name sglang_npu python=3.11 conda activate sglang_npu ``` #### CANN Prior to starting work with SGLang on Ascend, you need to install CANN Toolkit, Kernels operator package and NNAL version 9.0.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0008.html?OS=openEuler\&InstallType=local) #### MemFabric-Hybrid If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters. ```bash Command theme={null} pip install memfabric-hybrid==1.0.8 ``` #### MemFabric-zbal MemFabric-zbal is a Zero Buffer Acceleration Library of high-performance operators for LLM inference and training on Ascend, accelerating computation by eliminating intermediate memory buffers; it is required only on **aarch64** clusters and is installed in addition to MemFabric-Hybrid. ```bash Command theme={null} # Only needed on aarch64 (arm64) hosts pip install memfabric-zbal==1.1.1 ``` #### PyTorch and PyTorch Framework Adapter on Ascend ```bash Command theme={null} PYTORCH_VERSION=2.10.0 TORCHVISION_VERSION=0.25.0 TORCH_NPU_VERSION=2.10.0 pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu pip install torch_npu==$TORCH_NPU_VERSION ``` If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md) #### Triton on Ascend We provide our own implementation of Triton for Ascend. ```bash Command theme={null} pip install triton-ascend==3.2.1.dev20260530 \ --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \ --trusted-host mirrors.huaweicloud.com ``` For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://github.com/triton-lang/triton-ascend/blob/main/docs/en/installation_guide.md) #### SGLang Kernels NPU We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md). #### DeepEP-compatible Library We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md). #### Some other dependencies ```bash Command theme={null} # libGL apt update apt install libgl1 libglib2.0-0 # ensure setuptools contains pkg_resources module pip install "setuptools<80" ``` #### Installing SGLang from source ```bash Command theme={null} # Use the last release branch git clone https://github.com/sgl-project/sglang.git cd sglang mv python/pyproject_npu.toml python/pyproject.toml pip install -e python[all_npu] ``` ### Method 2: Using Docker Image #### Obtain Image You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. 1. Download SGLang image We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `cann9.0.0-a3-v0.5.16`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. ```bash Command theme={null} # Stable release docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 # Daily build docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 ``` ```bash Command theme={null} # Stable release docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 # Daily build docker pull quay.io/ascend/sglang:main-cann9.0.0-910b ``` 2. Build an image based on Dockerfile ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image # Replace with the target architecture, e.g., amd64, arm64. # Optional build arguments: # --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2 # --build-arg APTMIRROR= # Use a custom APT mirror to improve download speed # If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV. docker build --build-arg TARGETARCH= -t -f npu.Dockerfile . ``` #### Create Docker **Notice:** `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. ```bash Command theme={null} # Create a shortcut 'drun' to launch a privileged Docker container alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/' # Add HF_TOKEN env for downloading models by SGLang. # The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C) drun --env "HF_TOKEN=" \ \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend ``` ```bash Command theme={null} # Create a shortcut 'drun' to launch a privileged Docker container alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/' # Add HF_TOKEN env for downloading models by SGLang. # The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C) drun --env "HF_TOKEN=" \ \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend ``` SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters. ## System Settings ### CPU performance power scheme The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended. ```bash Command theme={null} echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor # Make sure changes are applied successfully cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance ``` ### Disable NUMA balancing ```bash Command theme={null} sudo sysctl -w kernel.numa_balancing=0 # Check cat /proc/sys/kernel/numa_balancing # shows 0 ``` ### Prevent swapping out system memory ```bash Command theme={null} sudo sysctl -w vm.swappiness=10 # Check cat /proc/sys/vm/swappiness # shows 10 ``` ## Running SGLang Service ### Running Service For Large Language Models #### PD Mixed Scene ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --attention-backend ascend \ --host 127.0.0.1 \ --port 30000 ``` #### PD Disaggregation Scene 1. Launch Prefill Server ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --device npu \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30001 ``` ```bash Command theme={null} # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --device npu \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30001 ``` 2. Launch Decode Server ```bash Command theme={null} # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --device npu \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30002 ``` ```bash Command theme={null} # PREFILL_IP: IP address of the first Prefill Server # FREE_PORT: any available port # all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT" export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma" python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --device npu \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ --port 30002 ``` 3. Launch Router ```bash Command theme={null} python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://127.0.0.1:30001 8995 \ --decode http://127.0.0.1:30002 \ --host 127.0.0.1 \ --port 30000 ``` The `8995` in command script is the disaggregation bootstrap port. It must match the `--disaggregation-bootstrap-port` value set on the prefill server in step 1. ### Running Service For Multimodal Language Models #### PD Mixed Scene ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \ --host 127.0.0.1 \ --port 30000 \ --tp 4 \ --device npu \ --attention-backend ascend \ --mm-attention-backend ascend_attn \ --disable-radix-cache \ --trust-remote-code \ --enable-multimodal \ --sampling-backend ascend ``` ## Testing the Service Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. ### Which port to send requests to The port you use depends on your deployment mode: | Scenario | Where to send requests | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Non-PD (single server) | The server's `--port` (e.g., `30000` in the examples above) | | Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes | | PD disaggregation | The router's `--port` (e.g., `30000` in the examples above); do **not** send requests directly to prefill or decode servers | SGLang defaults to port `30000` when `--port` is not specified. The examples in this guide use explicit ports for clarity. ### Health Check ```bash Command theme={null} curl http://127.0.0.1:30000/health ``` A successful response returns HTTP 200 with an empty body. ### Generate (Native Endpoint) ```bash Command theme={null} curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": {"temperature": 0, "max_new_tokens": 128} }' ``` The expected output should contain "Paris". ### Chat Completions (OpenAI-Compatible) ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "What is the capital of France?"}] }' ``` Some models return responses accompanied by thinking process content. To disable this output, configure parameters as follows: ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp", "messages": [{"role": "user", "content": "What is the capital of France?"}], "chat_template_kwargs": {"enable_thinking": false} }' ``` The expected output should contain "Paris". ### Multimodal Chat Completions The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data. ```bash Command theme={null} curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-VL-30B-A3B-Instruct", "messages": [{ "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] }] }' ``` # Quickstart Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/quick_start This page covers only the simplest deployment flow using the official container image. For the complete installation guide across all scenarios (source install, Docker build, system settings, version mapping, etc.), see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus/getting-started/installation). ## Prerequisites ### Supported Devices * Atlas 800I A2 inference series (Atlas 800I A2) * Atlas 800I A3 inference series (Atlas 800I A3) To identify your device, run `npu-smi info -l`: A3 reports `Chip Count: 2` per NPU, while A2 reports `Chip Count: 1` per NPU. For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ### Docker Ensure Docker is installed and the Docker daemon is running on your host machine. Verify with: ```bash theme={null} docker --version && docker info ``` If Docker is not installed, follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for your operating system. ## Setup environment using container Ensure sufficient disk space before proceeding. Run `df -h` to check the available disk space. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `cann9.0.0-a3-v0.5.16`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes. If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by adding `--volume /path/to/model:/path/to/model` to the `docker run` command below. ```shell Command theme={null} # Choose one (uncomment the line you want): export IMAGE=quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 # Stable release # export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ```shell Command theme={null} # Choose one (uncomment the line you want): export IMAGE=quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 # Stable release # export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b # Daily build docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --entrypoint=bash \ $IMAGE ``` ## Usage The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version. ### Start SGLang server SGLang will automatically download the model from Hugging Face. If the model is already downloaded to a local path (and has been mounted into the container), use that path directly like `--model-path /path/to/model`. ```shell Command theme={null} # Set HF_ENDPOINT to a mirror site if network is not available export HF_ENDPOINT=https://hf-mirror.com # Set your own HF_TOKEN to download restricted models export HF_TOKEN= # Start SGLang server # It may take several minutes to download the model on the first run sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend & ``` Server startup may take several minutes. Once you see output like the following, the server is running. ```log Output theme={null} INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit) The server is fired up and ready to roll! ``` ### Send a test request You can do inference using the server: ```shell Command theme={null} curl -X POST http://localhost:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "The capital of France is", "sampling_params": { "temperature": 0, "max_new_tokens": 16 } }' ``` If the "text" field in the response contains "Paris", the server is working as expected. ### Stop server and exit container The SGLang server is running as a background process. You can send a `SIGINT` signal to stop it. ```shell Command theme={null} SGLANG_PID=$(pgrep -f "sglang serve") kill -SIGINT $SGLANG_PID ``` Wait a moment for the server to shut down gracefully. The output should be like the following: ```log Output theme={null} INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [] ``` The server has now stopped. You can verify it with `ps -ef | grep sglang` — the expected output is nothing (no matching process), then exit the container by pressing `Ctrl+D`. # Mindspore backend Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/mindspore_backend ## Introduction MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang. ## Requirements MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages. The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2. ## Supported Models Currently, the following models are supported: * **Qwen3**: Dense and MoE models * **DeepSeek V3/R1** * *More models coming soon...* ## Installation Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./getting-started/installation) and then install `sgl-mindspore`: ```shell Install theme={null} git clone https://github.com/mindspore-lab/sgl-mindspore.git cd sgl-mindspore pip install -e . ``` ## Run Model Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. ### Offline infer Use the following script for offline infer: ```python Offline Inference theme={null} import sglang as sgl # Initialize the engine with MindSpore backend llm = sgl.Engine( model_path="/path/to/your/model", # Local model path device="npu", # Use NPU device model_impl="mindspore", # MindSpore implementation attention_backend="ascend", # Attention backend tp_size=1, # Tensor parallelism size dp_size=1 # Data parallelism size ) # Generate text prompts = [ "Hello, my name is", "The capital of France is", "The future of AI is" ] sampling_params = {"temperature": 0, "top_p": 0.9} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}") print(f"Generated: {output['text']}") print("---") ``` ### Start server Launch a server with MindSpore backend: ```bash Launch Server theme={null} # Basic server startup python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --tp-size 1 \ --dp-size 1 ``` For distributed server with multiple nodes: ```bash Multi-node Distributed theme={null} # Multi-node distributed server python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --dist-init-addr 127.0.0.1:29500 \ --nnodes 2 \ --node-rank 0 \ --tp-size 4 \ --dp-size 2 ``` ## Troubleshooting #### Debug Mode Enable sglang debug logging by log-level argument. ```bash Debug Mode theme={null} python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --log-level DEBUG ``` Enable mindspore info and debug logging by setting environments. ```bash Set Log Level theme={null} export GLOG_v=1 # INFO export GLOG_v=0 # DEBUG ``` #### Explicitly select devices Use the following environment variable to explicitly select the devices to use. ```shell Select Devices theme={null} export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device ``` #### Some communication environment issues In case of some environment with special communication environment, users need set some environment variables. ```shell Disable LCCL theme={null} export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore ``` #### Some dependencies of protobuf In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch. ```shell Fix Protobuf theme={null} export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch ``` ## Support For MindSpore-specific issues: * Refer to the [MindSpore documentation](https://www.mindspore.cn/) # DeepSeek-R1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1 This page focuses on optimal configuration and benchmark results for DeepSeek-R1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-R1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ----------- | ------------- | ----- | ----------------- | --------- | ------ | ------------ | ---------------------------------------------------------------------- | | DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 16ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-16ms) | | DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | | DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | | DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ----------- | ------------- | ----- | ----------------- | --------- | ---- | ------------ | --------------------------------------------------------------------- | | DeepSeek-R1 | Atlas 800I A3 | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | | DeepSeek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | | DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | ## Optimal Configuration ### DeepSeek-R1 W4A8 1P1D 16P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ENABLE_MOE_NZ=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=3500 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.62 \ --quantization modelslim \ --max-running-requests 32 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 20480 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 8 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --nnodes 1 \ --tp-size 16 \ --dp-size 16 \ --mem-fraction-static 0.805 \ --max-running-requests 416 \ --quantization modelslim \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --tokenizer-worker-num 4 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 416 \ --random-input-len 3584 \ --random-output-len 1536 \ --num-prompts 1664 \ --random-range-ratio 1 \ --request-rate 24 ``` ### DeepSeek-R1 W4A8 8P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=56 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --watchdog-timeout 9000 \ --cuda-graph-bs 4 8 12 14 \ --mem-fraction-static 0.77 \ --max-running-requests 224 \ --context-length 8188 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 3000 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --enable-dp-attention \ --dp-size 16 \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --dtype bfloat16 \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 224 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 896 \ --random-range-ratio 1 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 16ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 16ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1536 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 2 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 32 \ --random-range-ratio 1 \ --request-rate 16 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 50ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=131072 export SGLANG_USE_AG_AFTER_QLORA=1 export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.778 \ --max-running-requests 16 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 60000 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 4 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=600 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_LM_HEAD_TP=8 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --dp-size 32 \ --mem-fraction-static 0.82 \ --max-running-requests 1024 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 1 \ --enable-dp-attention \ --deepep-mode low_latency \ --moe-dense-tp 1 \ --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1024 \ --random-input-len 3584 \ --random-output-len 1536 \ --num-prompts 7168 \ --random-range-ratio 1 \ --request-rate 40 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K 19.0ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1k **TPOT**: 19.0ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1536 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 2 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 3500 \ --random-output-len 1024 \ --num-prompts 32 \ --random-range-ratio 1 \ --request-rate 16 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN3K9 OUT1K 19.0ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 3.9k+1k **TPOT**: 19.0ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1536 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --context-length 8192 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 2 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 3900 \ --random-output-len 1024 \ --num-prompts 32 \ --random-range-ratio 1 \ --request-rate 16 ``` ### DeepSeek-R1 W8A8 2P1D 32P IN6K OUT1K6 20.5ms **Model**: DeepSeek-R1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 6k+1.6k **TPOT**: 20.5ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_NPU_USE_MLAPO=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_USE_FIA_NZ=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1536 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --nnodes 1 \ --tp-size 16 \ --mem-fraction-static 0.81 \ --quantization modelslim \ --max-running-requests 4 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 28680 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --dp-size 2 \ --enable-dp-attention \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --enable-attn-tp-input-scattered \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp-size 32 \ --dp-size 8 \ --mem-fraction-static 0.75 \ --max-running-requests 32 \ --quantization modelslim \ --moe-a2a-backend deepep \ --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs 2 4 6 \ --watchdog-timeout 9000 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser deepseek-r1 \ --tool-call-parser deepseekv3 \ --disaggregation-transfer-backend ascend \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ export SGLANG_DP_ROUND_ROBIN=1 python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 6000 \ --random-output-len 1600 \ --num-prompts 32 \ --random-range-ratio 1 \ --request-rate 16 ``` # DeepSeek-V3.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2 This page focuses on optimal configuration and benchmark results for DeepSeek-V3.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V3.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------------- | ------- | ---- | ------------ | ----------------------------------------------------------------------- | | DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | | DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------------- | ------- | ----- | ------------ | ----------------------------------------------------------------------- | | DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | ## Optimal Configuration ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K 26ms **Model**: DeepSeek-V3.2 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 26ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --dp 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-attention \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 8 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 8 \ --random-range-ratio 1 ``` ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS16 **Model**: DeepSeek-V3.2 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 107ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --dp 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-attention \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 16 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 16 \ --random-range-ratio 1 ``` ### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS8 **Model**: DeepSeek-V3.2 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 26ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.73 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --disable-cuda-graph \ --moe-dense-tp-size 1 \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 32 \ --speculative-algorithm NEXTN \ --speculative-num-steps 1 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --nnodes 2 \ --tp 32 \ --dp 8 \ --ep 32 \ --moe-dense-tp-size 1 \ --enable-dp-attention \ --enable-dp-lm-head \ --watchdog-timeout 9000 \ --mem-fraction-static 0.79 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 68000 \ --max-running-requests 32 \ --cuda-graph-max-bs-decode 4 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --quantization modelslim \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --disaggregation-transfer-backend ascend \ --reasoning-parser deepseek-v3 \ --tool-call-parser deepseekv32 \ --trust-remote-code \ --attention-backend ascend \ --device npu NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 8 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 8 \ --random-range-ratio 1 ``` # GLM-5.1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1 This page focuses on optimal configuration and benchmark results for GLM-5.1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | ------- | ------------- | ----- | ----------------- | ------------------------------------ | ---- | ---- | ------------ | -------------------------------------------------------------------------- | | GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | ------- | ------------- | ----- | ----------------- | ------------------------------------- | ------ | ----- | ------------ | --------------------------------------------------------------------------- | | GLM-5.1 | Atlas 800I A3 | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | | GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | | GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | | GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | | GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | | GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 65k+1.5k (100% prefix cache hit rate) | 33ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-48p-in65k-out1k5-prefix100-33ms) | | GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | | GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 64k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration ### GLM-5.1 W4A8 16P IN3K5 OUT1K5 50ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # NODE_IPS: IP addresses of each node in the cluster # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights NODE_IPS=('' '') echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2500 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" for i in "${!NODE_IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]]; then echo "${NODE_IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host ${NODE_IPS[$i]} --port 6688 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --attention-backend ascend \ --device npu \ --tp-size 32 \ --dp-size 16 \ --enable-dp-attention \ --chunked-prefill-size 65536 \ --max-prefill-tokens 280000 \ --trust-remote-code \ --mem-fraction-static 0.65 \ --served-model-name glm-5 \ --cuda-graph-max-bs-decode 16 \ --max-running-requests 256 \ --quantization modelslim \ --speculative-draft-model-quantization unquant \ --moe-a2a-backend deepep \ --deepep-mode auto \ --load-balance-method round_robin \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 break fi done ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 128 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 128 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN128K OUT1K 56.4ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 56.4ms **TTFT**: 13.1s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 8192 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN16K OUT1K 50ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --mem-fraction-static 0.75 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 64 \ --served-model-name glm-5 \ --chunked-prefill-size 524288 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --dp-size 4 \ --enable-dp-attention \ --load-balance-method round_robin \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 8 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --mem-fraction-static 0.87 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 128 \ --random-input-len 16384 \ --random-output-len 1024 \ --num-prompts 512 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 55.2ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 55.2ms **TTFT**: 7.58s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 8192 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --enable-dp-attention \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 65536 \ --random-output-len 1024 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 50ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --enable-dp-attention \ --ep-size 32 \ --mem-fraction-static 0.85 \ --max-running-requests 32 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 65536 \ --random-output-len 1024 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 1P1D 32P IN65K OUT1K5 PREFIX90 25ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 32 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 65k+1.5k (90% prefix cache hit rate) **TPOT**: 25ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --mem-fraction-static 0.75 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 64 \ --served-model-name glm-5 \ --chunked-prefill-size 524288 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --dp-size 4 \ --enable-dp-attention \ --load-balance-method round_robin \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 8 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --mem-fraction-static 0.87 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 59904` = `int(66560 * 0.9)` is the shared prefix portion. `--gsp-question-len 6656` = `int(66560 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 480 \ --gsp-system-prompt-len 59904 \ --gsp-question-len 6656 \ --gsp-output-len 1536 \ --max-concurrency 100 \ --num-prompts 480 \ --request-rate inf ``` ### GLM-5.1 W4A8 1P1D 48P IN65K OUT1K5 PREFIX100 33ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 48 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 65k+1.5k (100% prefix cache hit rate) **TPOT**: 33ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '') D_IP=('' '' '' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[0]}:5000 \ --disaggregation-bootstrap-port 8998 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 192 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=650 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=48 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 64 \ --nnodes 4 \ --dp-size 64 \ --ep-size 64 \ --enable-dp-attention \ --mem-fraction-static 0.84 \ --max-running-requests 192 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --cuda-graph-bs 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 128 \ --random-input-len 66560 \ --random-output-len 1536 \ --num-prompts 512 \ --random-range-ratio 1 ``` ### GLM-5.1 W4A8 2P1D 48P IN128K OUT1K PREFIX90 50ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 48 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '' '' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --dist-init-addr ${P_IP[$(( $i / 2 * 2 ))]}:5000 \ --disaggregation-bootstrap-port $((8998 + $i / 2)) \ --node-rank $(( $i % 2 )) \ --tp-size 4 \ --nnodes 2 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 32 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 8 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --mem-fraction-static 0.865 \ --max-running-requests 96 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 4 5 6 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 32 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117964` = `int(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `int(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 576 \ --gsp-system-prompt-len 117964 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 144 \ --num-prompts 576 \ --request-rate inf ``` ### GLM-5.1 W4A8 4P1D 48P IN64K OUT1K PREFIX90 50ms **Model**: GLM-5.1 **Hardware**: Atlas 800I A3 **Cards**: 48 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('' '' '' '') D_IP=('' '') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export ENABLE_PROFILING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export TASK_QUEUE_ENABLE=2 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port $((8998 + $i)) \ --node-rank 0 \ --tp-size 4 \ --nnodes 1 \ --mem-fraction-static 0.72 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --disaggregation-transfer-backend ascend \ --max-running-requests 16 \ --served-model-name glm-5 \ --chunked-prefill-size 16384 \ --max-prefill-tokens 180000 \ --moe-a2a-backend deepep \ --deepep-mode normal \ --disable-shared-experts-fusion \ --disable-cuda-graph \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --enable-nsa-prefill-context-parallel \ --nsa-prefill-cp-mode in-seq-split \ --attn-cp-size 4 \ --enable-dp-lm-head \ --moe-dense-tp 1 \ --pp-size 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=300 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --dist-init-addr ${D_IP[0]}:5000 \ --node-rank $i \ --tp-size 32 \ --nnodes 2 \ --dp-size 32 \ --ep-size 32 \ --enable-dp-attention \ --mem-fraction-static 0.85 \ --max-running-requests 320 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser glm45 \ --tool-call-parser glm47 \ --trust-remote-code NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # , , , : prefill node IP addresses # : first decode node IP address (decode may have distributed nodes) # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --prefill http://:8000 8999 \ --prefill http://:8000 9000 \ --prefill http://:8000 9001 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy round_robin ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `int(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6553` = `int(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 1280 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6553 \ --gsp-output-len 1024 \ --max-concurrency 320 \ --num-prompts 1280 \ --request-rate inf ``` # Kimi-K2.6 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6 This page focuses on optimal configuration and benchmark results for Kimi-K2.6 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Kimi-K2.6 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --------- | ------------- | ----- | ----------- | --------- | ---- | ---- | ------------ | ------------------------------------------------------------- | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --------- | ------------- | ----- | ----------------- | ------------------------------------ | ----- | ---- | ------------ | ----------------------------------------------------------------------------- | | Kimi-K2.6 | Atlas 800I A3 | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1080p\_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Kimi-K2.6 W4A8 16P IN64K OUT1K 100ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # NODE_IPS: IP addresses of each node in the cluster # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights NODE_IPS=('' '') echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=4400 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" for i in "${!NODE_IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]]; then echo "${NODE_IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host ${NODE_IPS[$i]} --port 6688 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 32 \ --mem-fraction-static 0.662 \ --max-running-requests 32 \ --chunked-prefill-size 262144 \ --context-length 75000 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --enable-dp-attention \ --dp-size 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 1 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 break fi done ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 64000 \ --random-output-len 1000 \ --num-prompts 32 \ --random-range-ratio 1 ``` ### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K 100ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --disable-radix-cache \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --enable-dp-attention \ --dp-size 1 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 128000 \ --random-output-len 1000 \ --num-prompts 1 \ --random-range-ratio 1 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K PREFIX90 100ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 100ms **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=8 export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699 export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --enable-dp-attention \ --dp-size 1 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 115200` = `int(128000 * 0.9)` is the shared prefix portion. `--gsp-question-len 12800` = `int(128000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 4 \ --gsp-system-prompt-len 115200 \ --gsp-question-len 12800 \ --gsp-output-len 1000 \ --max-concurrency 1 \ --num-prompts 4 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 100ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1.5k **TPOT**: 100ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1800 export HCCL_SOCKET_IFNAME= python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --disable-radix-cache \ --disable-cuda-graph \ --mem-fraction-static 0.78 \ --max-running-requests 1 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 1 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 16 \ --enable-dp-attention \ --dp-size 1 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 16 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 64000 \ --random-output-len 1500 \ --num-prompts 1 \ --random-range-ratio 1 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 PREFIX90 100ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 16 **Deploy Mode**: PD Disaggregation **Quantization**: W4A8 INT8 **Dataset**: 64k+1.5k (90% prefix cache hit rate) **TPOT**: 100ms **TTFT**: 3s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1800 export HCCL_SOCKET_IFNAME= python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.78 \ --max-running-requests 2 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --chunked-prefill-size 16384 \ --prefill-max-requests 2 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --quantization modelslim \ --dtype bfloat16 \ --disaggregation-transfer-backend ascend \ --nnodes 1 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --mem-fraction-static 0.82 \ --max-running-requests 2 \ --enable-dp-attention \ --dp-size 2 \ --enable-dp-lm-head \ --disable-radix-cache \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 1 2 4 6 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --policy cache_aware ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion. `--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 8 \ --gsp-system-prompt-len 57600 \ --gsp-question-len 6400 \ --gsp-output-len 1500 \ --max-concurrency 2 \ --num-prompts 8 \ --request-rate inf ``` ### Kimi-K2.6 W4A8 8P IN1024X1024 30 OUT1024 50ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_BUFFSIZE=1500 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --quantization modelslim \ --dtype bfloat16 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --trust-remote-code \ --device npu \ --attention-backend ascend \ --tp-size 16 \ --mem-fraction-static 0.872 \ --max-running-requests 176 \ --chunked-prefill-size 32768 \ --context-length 8192 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --enable-dp-attention \ --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 9 10 11 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1024x1024 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 160 \ --random-input-len 30 \ --random-output-len 1024 \ --num-prompts 640 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 16 \ --image-count 1 \ --image-resolution 1024x1024 ``` ### Kimi-K2.6 W4A8 8P IN1080P 30 OUT256 50ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_BUFFSIZE=2400 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --quantization modelslim \ --dtype bfloat16 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --trust-remote-code \ --device npu \ --attention-backend ascend \ --tp-size 16 \ --base-gpu-id 0 \ --mem-fraction-static 0.852 \ --max-running-requests 64 \ --chunked-prefill-size 16384 \ --context-length 8192 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --enable-dp-attention \ --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `IMAGE` dataset with 1920x1080 resolution. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name image \ --backend sglang-oai-chat \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 48 \ --random-input-len 30 \ --random-output-len 256 \ --num-prompts 196 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 16 \ --image-count 1 \ --image-resolution 1920x1080 ``` ### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 20ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 16 \ --mem-fraction-static 0.865 \ --max-running-requests 80 \ --chunked-prefill-size 32768 \ --context-length 6144 \ --max-prefill-tokens 65536 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --enable-dp-attention \ --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 3 4 5 \ --disable-radix-cache \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 256 \ --random-range-ratio 1 \ --warmup-requests 0 ``` ### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 50ms **Model**: Kimi-K2.6 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1200 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 16 \ --mem-fraction-static 0.895 \ --max-running-requests 208 \ --chunked-prefill-size 32768 \ --context-length 6144 \ --max-prefill-tokens 16384 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --sampling-backend ascend \ --enable-dp-attention \ --dp-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs-decode 1 2 4 8 12 13 \ --disable-radix-cache \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --speculative-draft-model-quantization unquant \ --prefill-delayer-max-delay-passes 200 \ --enable-prefill-delayer \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 192 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 768 \ --random-range-ratio 1 \ --warmup-requests 0 ``` # MiMo-V2-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash This page focuses on optimal configuration and benchmark results for MiMo-V2-Flash on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiMo-V2-Flash Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------------- | ------- | ---- | ---- | ------------ | ---------------------------------------------------------------------- | | MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | | MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------------- | ------- | ---- | ---- | ------------ | ------------------------------------------------------------------- | | MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | | MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | ## Optimal Configuration ### MiMo-V2-Flash 1P1D 12P IN16K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Atlas 800I A3 **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 16k+1 **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --disable-piecewise-cuda-graph \ --dp-size 2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --random-input-len 16000 \ --random-output-len 1 \ --num-prompts 128 \ --random-range-ratio 1 \ --request-rate 0.4 ``` ### MiMo-V2-Flash 1P1D 12P IN16K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Atlas 800I A3 **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 16k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --disable-piecewise-cuda-graph \ --dp-size 2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 32 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs 1 2 4 8 12 16 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 32 \ --random-input-len 16000 \ --random-output-len 1000 \ --num-prompts 128 \ --random-range-ratio 1 \ --request-rate inf ``` ### MiMo-V2-Flash 1P1D 12P IN32K OUT1 TTFT 5s **Model**: MiMo-V2-Flash **Hardware**: Atlas 800I A3 **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 32k+1 **TTFT**: 5s #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --disable-piecewise-cuda-graph \ --dp-size 2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --random-input-len 32000 \ --random-output-len 1 \ --num-prompts 128 \ --random-range-ratio 1 \ --request-rate 0.4 ``` ### MiMo-V2-Flash 1P1D 12P IN32K OUT1K TPOT 20ms **Model**: MiMo-V2-Flash **Hardware**: Atlas 800I A3 **Cards**: 12 **Deploy Mode**: PD Disaggregation **Quantization**: W8A8 INT8 **Dataset**: 32k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # P_IP: prefill node IP address # D_IP: decode node IP address # ASCEND_MF_STORE_URL: prefill node IP with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export HCCL_CONNECT_TIMEOUT=1800 export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 P_IP=('') D_IP=('') export ASCEND_MF_STORE_URL="tcp://:24670" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --host ${P_IP[$i]} \ --port 8000 \ --disaggregation-bootstrap-port 8998 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --tp-size 8 \ --nnodes 1 \ --chunked-prefill-size 8192 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --disable-piecewise-cuda-graph \ --dp-size 2 NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=800 export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --host ${D_IP[$i]} \ --port 8001 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --nnodes 1 \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ --disable-radix-cache \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode low_latency NODE_RANK=$i break fi done ``` ```bash Command theme={null} # ============================================================ # Before running, replace the following placeholders: # : prefill node IP address # : decode node IP address # ============================================================ python -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8998 \ --decode http://:8001 \ --host 127.0.0.1 \ --port 6688 \ --health-check-interval-secs 3600 --mini-lb ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --random-input-len 32000 \ --random-output-len 1000 \ --num-prompts 128 \ --random-range-ratio 1 \ --request-rate inf ``` # MiniMax-M2.5 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5 This page focuses on optimal configuration and benchmark results for MiniMax-M2.5 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiniMax-M2.5 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------ | ------------- | ----- | ----------- | ----------------------------------- | ------- | ------------ | ---------------------------------------------------------------------------- | | MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | | MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------ | ------------- | ----- | ----------- | ---------------------------------- | ---- | ------------ | ------------------------------------------------------------------------ | | MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | | MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | | MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration ### MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms **Model**: MiniMax-M2.5 **Hardware**: Atlas 800I A3 **Cards**: 4 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 32k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=128 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60184 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBAL_ENABLE_GRAPH=1 export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 8 \ --disable-radix-cache \ --mem-fraction-static 0.74 \ --max-running-requests 18 \ --chunked-prefill-size -1 \ --max-prefill-tokens 32768 \ --cuda-graph-bs 2 4 6 8 10 12 14 16 18 24 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --trust-remote-code \ --tokenizer-worker-num 4 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 18 \ --random-input-len 32768 \ --random-output-len 1024 \ --num-prompts 72 \ --random-range-ratio 1 ``` ### MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms **Model**: MiniMax-M2.5 **Hardware**: Atlas 800I A3 **Cards**: 4 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=140000 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 8 \ --mem-fraction-static 0.63 \ --max-running-requests 26 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --enable-prefill-delayer \ --prefill-max-requests 10 \ --chunked-prefill-size 67072 \ --max-prefill-tokens 67000 \ --cuda-graph-bs 2 4 8 12 16 18 20 22 24 26 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 \ --trust-remote-code ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `int(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6553` = `int(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 104 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6553 \ --gsp-output-len 1024 \ --max-concurrency 26 \ --num-prompts 104 \ --request-rate inf ``` ### MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 24.44ms **Model**: MiniMax-M2.5 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 24.44ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --dp-size 2 \ --enable-dp-attention \ --mem-fraction-static 0.65 \ --max-running-requests 4 \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --enable-prefill-delayer \ --prefill-max-requests 4 \ --chunked-prefill-size 160000 \ --max-prefill-tokens 80000 \ --cuda-graph-bs 2 4 6 8 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --tokenizer-worker-num 4 \ --dtype bfloat16 ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117964` = `int(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `int(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 16 \ --gsp-system-prompt-len 117964 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 4 \ --num-prompts 16 \ --request-rate inf ``` ### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms **Model**: MiniMax-M2.5 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2048 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --enable-dp-attention \ --dp-size 16 \ --mem-fraction-static 0.53 \ --max-running-requests 96 \ --disable-radix-cache \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --prefill-delayer-max-delay-passes 500 \ --enable-prefill-delayer \ --prefill-max-requests 3 \ --chunked-prefill-size -1 \ --max-prefill-tokens 8192 \ --cuda-graph-bs 1 2 3 4 5 6 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 112 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 448 \ --random-range-ratio 1 ``` ### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms **Model**: MiniMax-M2.5 **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1024 export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 16 \ --enable-dp-attention \ --dp-size 16 \ --mem-fraction-static 0.75 \ --max-running-requests 320 \ --disable-radix-cache \ --reasoning-parser minimax-append-think \ --tool-call-parser minimax-m2 \ --prefill-delayer-max-delay-passes 500 \ --enable-prefill-delayer \ --chunked-prefill-size -1 \ --max-prefill-tokens 8192 \ --cuda-graph-bs 1 2 4 8 12 16 20 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --deepep-mode auto \ --quantization modelslim \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --dtype bfloat16 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 320 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 1280 \ --random-range-ratio 1 ``` # Qwen3-235B-A22B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b This page focuses on optimal configuration and benchmark results for Qwen3-235B-A22B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-235B-A22B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------- | ------------- | ----- | ----------- | -------- | ---- | ------------ | ------------------------------------------------------------------ | | Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------- | ------------- | ----- | ----------- | --------- | ------ | ------------ | --------------------------------------------------------------------- | | Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | ## Optimal Configuration ### Qwen3-235B-A22B BF16 8P IN11K OUT1K5 8ms **Model**: Qwen3-235B-A22B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 11k+1.5k **TPOT**: 8ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 1 \ --dtype bfloat16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 16384 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --disable-radix-cache \ --enable-dp-lm-head \ --tp 16 \ --mem-fraction-static 0.78 \ --cuda-graph-bs 1 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 11000 \ --random-output-len 1500 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### Qwen3-235B-A22B W8A8 8P IN3K5 OUT1K5 50.1ms **Model**: Qwen3-235B-A22B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50.1ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=570 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 432 \ --context-length 8192 \ --dtype bfloat16 \ --chunked-prefill-size 94208 \ --max-prefill-tokens 458880 \ --sampling-backend ascend \ --ep-dispatch-algorithm static \ --disable-radix-cache \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --tp 16 \ --dp-size 16 \ --enable-dp-attention \ --enable-dp-lm-head \ --mem-fraction-static 0.8 \ --cuda-graph-bs 1 2 4 8 16 20 24 26 27 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 432 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 1728 \ --random-range-ratio 1 ``` # Qwen3-30B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b This page focuses on optimal configuration and benchmark results for Qwen3-30B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-30B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------- | --------- | ------- | ------------ | ----------------------------------------------------------------- | | Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | | Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------- | ------------- | ----- | ----------- | --------- | ------- | ------------ | ----------------------------------------------------------------- | | Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | | Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Qwen3-30B-A3B BF16 1P IN1K OUT100 **Model**: Qwen3-30B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1k+100 **TPOT**: 10000ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export DP_ROUND_ROBIN=1 export GLOO_SOCKET_IFNAME= export HCCL_ALGO="level0:NA;level1:ring" export HCCL_SOCKET_IFNAME= export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_USE_MAX_DP_ATT=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 168 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --max-prefill-tokens 8300 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 7 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 8 \ --tp-size 2 \ --enable-dp-attention \ --dp-size 2 \ --mem-fraction-static 0.85 \ --cuda-graph-bs 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 162 \ --random-input-len 1000 \ --random-output-len 100 \ --num-prompts 624 \ --random-range-ratio 1 ``` ### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 10ms **Model**: Qwen3-30B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 10ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 162 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ --cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 50ms **Model**: Qwen3-30B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_LAUNCH_BLOCKING=0 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 162 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ --cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 160 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 640 \ --random-range-ratio 1 ``` ### Qwen3-30B-A3B W8A8 1P IN6K OUT1K5 BS16 **Model**: Qwen3-30B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 10.25ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=400 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export TRANSFORMERS_VERBOSITY=error python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 16 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --tp-size 2 \ --mem-fraction-static 0.6 \ --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --num-prompts 16 \ --random-range-ratio 1 ``` # Qwen3-32B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b This page focuses on optimal configuration and benchmark results for Qwen3-32B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-32B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------- | ------------- | ----- | ----------- | ------- | ---- | ------------ | ----------------------------------------------------------- | | Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------- | ------------- | ----- | ----------- | --------- | ---- | ------------ | ---------------------------------------------------------------- | | Qwen3-32B | Atlas 800I A2 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms-a2) | | Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Qwen3-32B BF16 8P IN18K OUT4K 6ms **Model**: Qwen3-32B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 18k+4k **TPOT**: 6ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --max-running-requests 1 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --tp-size 16 \ --mem-fraction-static 0.72 \ --cuda-graph-bs 1 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 18000 \ --random-output-len 4000 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### Qwen3-32B W8A8 2P IN3K5 OUT1K5 50ms A2 **Model**: Qwen3-32B **Hardware**: Atlas 800I A2 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_DEEPGEMM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 101 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ --cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 100 \ --random-input-len 3584 \ --random-output-len 1536 \ --num-prompts 400 \ --random-range-ratio 1 ``` ### Qwen3-32B W8A8 2P IN3K5 OUT1K5 50ms **Model**: Qwen3-32B **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_DEEPGEMM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 101 \ --disable-radix-cache \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ --cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 100 \ --random-input-len 3584 \ --random-output-len 1536 \ --num-prompts 400 \ --random-range-ratio 1 ``` # Qwen3.5-397B-A17B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b This page focuses on optimal configuration and benchmark results for Qwen3.5-397B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.5-397B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------ | ------------- | ----- | ----------- | --------- | ---- | ------------ | ---------------------------------------------------------------- | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-20ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-20ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-20ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ------------ | ------------- | ----- | ----------- | ----------------------------------- | ---- | ------------ | ------------------------------------------------------------------------- | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-50ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-prefix90-50ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-50ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-50ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-50ms) | | Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration ### Qwen3.5-397B W4A8 8P IN128K OUT1K 20ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --cuda-graph-bs 2 3 4 5 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 3 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 3 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 2 ``` ### Qwen3.5-397B W4A8 8P IN128K OUT1K 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --cuda-graph-bs 2 4 6 8 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 10 \ --random-input-len 131072 \ --random-output-len 1024 \ --num-prompts 10 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 8 ``` ### Qwen3.5-397B W4A8 8P IN128K OUT1K PREFIX90 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2200 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 131072 \ --max-mamba-cache-size 320 \ --prefill-max-requests 10 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.6 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 117964` = `int(131072 * 0.9)` is the shared prefix portion. `--gsp-question-len 13107` = `int(131072 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 40 \ --gsp-system-prompt-len 117964 \ --gsp-question-len 13107 \ --gsp-output-len 1024 \ --max-concurrency 40 \ --num-prompts 40 \ --request-rate inf ``` ### Qwen3.5-397B W4A8 8P IN16K OUT1K 20ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 50000 \ --prefill-max-requests 4 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 48 \ --mem-fraction-static 0.8 \ --max-total-tokens 210000 \ --cuda-graph-bs 2 4 6 8 10 12 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 4 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 40 \ --random-input-len 16384 \ --random-output-len 1024 \ --num-prompts 40 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 32 ``` ### Qwen3.5-397B W4A8 8P IN16K OUT1K 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 4 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 144 \ --mem-fraction-static 0.8 \ --max-total-tokens 635000 \ --cuda-graph-bs 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 4 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 132 \ --random-input-len 16384 \ --random-output-len 1024 \ --num-prompts 132 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 8 ``` ### Qwen3.5-397B W4A8 8P IN3K5 OUT1K5 20ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=6 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 35000 \ --max-total-tokens 128000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 160 \ --mem-fraction-static 0.8 \ --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 8 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 160 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 160 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 64 ``` ### Qwen3.5-397B W4A8 8P IN3K5 OUT1K5 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 export DEEPEP_NORMAL_LONG_SEQ_ROUND=6 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 17500 \ --max-total-tokens 280000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 432 \ --mem-fraction-static 0.8 \ --cuda-graph-bs 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 8 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 432 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 432 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 16 ``` ### Qwen3.5-397B W4A8 8P IN64K OUT1K 20ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ --cuda-graph-bs 2 4 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 6 \ --random-input-len 65536 \ --random-output-len 1024 \ --num-prompts 6 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 6 ``` ### Qwen3.5-397B W4A8 8P IN64K OUT1K 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=0 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --prefill-max-requests 1 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 32 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ --cuda-graph-bs 2 4 6 8 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 28 \ --random-input-len 65536 \ --random-output-len 1024 \ --num-prompts 28 \ --random-range-ratio 1 \ --request-rate inf \ --warmup-requests 8 ``` ### Qwen3.5-397B W4A8 8P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.5-397B **Hardware**: Atlas 800I A3 **Cards**: 8 **Deploy Mode**: PD Mixed **Quantization**: W4A8 INT8 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=20 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2200 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --attention-backend ascend \ --device npu \ --tp-size 16 \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --max-mamba-cache-size 640 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 128 \ --mem-fraction-static 0.6 \ --max-total-tokens 1310720 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ --deepep-mode auto \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `int(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6553` = `int(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 112 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6553 \ --gsp-output-len 1024 \ --max-concurrency 112 \ --num-prompts 112 \ --request-rate inf ``` # Qwen3.6-27B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b This page focuses on optimal configuration and benchmark results for Qwen3.6-27B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-27B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | ----------- | ------------- | ----- | ----------- | ---------------------------------- | ---- | ------------ | -------------------------------------------------------------------- | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | ## Optimal Configuration ### Qwen3.6-27B 1P IN1024X1024 30 OUT1024 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=150 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 52000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 50 \ --max-mamba-cache-size 60 \ --mem-fraction-static 0.76 \ --cuda-graph-bs 2 4 8 16 24 32 40 42 45 50 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mm-enable-dp-encoder \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 48 \ --random-input-len 30 \ --random-output-len 1024 \ --num-prompts 48 \ --random-range-ratio 1 ``` ### Qwen3.6-27B 1P IN1080P 30 OUT256 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=150 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 48000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 30 \ --max-mamba-cache-size 40 \ --mem-fraction-static 0.76 \ --cuda-graph-bs 2 4 8 16 24 28 30 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --mm-enable-dp-encoder \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 30 \ --random-input-len 30 \ --random-output-len 256 \ --num-prompts 120 \ --random-range-ratio 1 ``` ### Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 20 \ --max-mamba-cache-size 160 \ --mem-fraction-static 0.82 \ --cuda-graph-bs 1 2 5 10 15 17 19 20 \ --enable-prefill-delayer \ --prefill-delayer-queue-min-ratio 0.7 \ --prefill-delayer-max-delay-ms 20000 \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion. `--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 80 \ --gsp-system-prompt-len 57600 \ --gsp-question-len 6400 \ --gsp-output-len 1000 \ --max-concurrency 20 \ --num-prompts 80 \ --request-rate inf ``` ### Qwen3.6-27B W8A8 1P IN3K5 OUT1K5 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=130 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 60000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 64 \ --max-mamba-cache-size 74 \ --mem-fraction-static 0.7 \ --cuda-graph-bs 2 8 16 32 40 45 50 54 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 54 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 216 \ --random-range-ratio 1 ``` ### Qwen3.6-27B W8A8 1P IN64K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 48000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 6 \ --max-mamba-cache-size 16 \ --mem-fraction-static 0.6 \ --cuda-graph-bs 1 2 4 5 6 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 6 \ --random-input-len 64000 \ --random-output-len 1000 \ --num-prompts 12 \ --random-range-ratio 1 ``` ### Qwen3.6-27B W8A8 2P IN128K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 74000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 6 \ --max-mamba-cache-size 7 \ --mem-fraction-static 0.63 \ --cuda-graph-bs 1 2 4 5 6 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 4 \ --random-input-len 128000 \ --random-output-len 1000 \ --num-prompts 16 \ --random-range-ratio 1 ``` ### Qwen3.6-27B W8A8 2P IN16K OUT1K 50ms **Model**: Qwen3.6-27B **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 16k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 58000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 29 \ --max-mamba-cache-size 58 \ --mem-fraction-static 0.68 \ --cuda-graph-bs 1 2 8 12 16 20 24 26 28 29 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 29 \ --random-input-len 16000 \ --random-output-len 1000 \ --num-prompts 116 \ --random-range-ratio 1 ``` # Qwen3.6-35B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b This page focuses on optimal configuration and benchmark results for Qwen3.6-35B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-35B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. Use image **SGLang `>= v0.5.16`** for these NEXTN configurations. Without `--dataset-path`, `bench_serving --dataset-name random` downloads ShareGPT from Hugging Face; in offline environments, pass a local dataset path (for example a ShareGPT JSON file). ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------- | ------------- | ----- | ----------- | ------- | ------ | ------------ | --------------------------------------------------------- | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------- | ------------- | ----- | ----------- | ---------------------------------- | ------- | ------------ | ------------------------------------------------------------------------ | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | ## Optimal Configuration ### Qwen3.6-35B-A3B 1P IN1024X1024 30 OUT1024 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1024x1024 (30)+1024 *Format: resolution (input tokens) + output tokens* **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --trust-remote-code \ --enable-prefill-delayer \ --max-running-requests 120 \ --max-mamba-cache-size 240 \ --mem-fraction-static 0.78 \ --cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 120 \ --random-input-len 30 \ --random-output-len 1024 \ --num-prompts 480 \ --random-range-ratio 1 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 1P IN1080P 30 OUT256 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 1080p\_30+256 **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=10 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --trust-remote-code \ --enable-prefill-delayer \ --max-running-requests 50 \ --max-mamba-cache-size 55 \ --mem-fraction-static 0.8 \ --cuda-graph-bs 2 4 8 12 16 20 24 28 32 36 40 44 48 50 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 50 \ --random-input-len 30 \ --random-output-len 256 \ --num-prompts 200 \ --random-range-ratio 1 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 1P IN128K OUT1K 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 128k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1600 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 520960 \ --max-prefill-tokens 128000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 3 \ --max-mamba-cache-size 10 \ --mem-fraction-static 0.9 \ --cuda-graph-bs 1 2 3 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 3 \ --random-input-len 128000 \ --random-output-len 1000 \ --num-prompts 3 \ --random-range-ratio 1 ``` ### Qwen3.6-35B-A3B 1P IN254K OUT1K **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 254k+1k **TPOT**: 16.1ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 131072 \ --max-prefill-tokens 254000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.65 \ --cuda-graph-bs 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 254000 \ --random-output-len 1000 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### Qwen3.6-35B-A3B 1P IN3K5 OUT1K5 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=1 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 659840 \ --max-prefill-tokens 43400 \ --disable-radix-cache \ --trust-remote-code \ --prefill-max-requests 12 \ --max-running-requests 122 \ --max-mamba-cache-size 122 \ --mem-fraction-static 0.9 \ --cuda-graph-bs 4 16 32 64 96 116 120 122 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 122 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 122 \ --random-range-ratio 1 ``` ### Qwen3.6-35B-A3B 1P IN64K OUT1K 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 600000 \ --max-prefill-tokens 65536 \ --disable-radix-cache \ --trust-remote-code \ --enable-prefill-delayer \ --max-running-requests 10 \ --max-mamba-cache-size 20 \ --mem-fraction-static 0.65 \ --cuda-graph-bs 2 4 8 12 14 16 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 10 \ --random-input-len 64000 \ --random-output-len 1000 \ --num-prompts 40 \ --random-range-ratio 1 ``` ### Qwen3.6-35B-A3B 1P IN64K OUT1K PREFIX90 50ms **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GDN_ATTN_BACKEND_TRITON=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=300 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 2 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size -1 \ --max-total-tokens 470784 \ --max-prefill-tokens 65536 \ --trust-remote-code \ --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 40 \ --max-mamba-cache-size 200 \ --mem-fraction-static 0.9 \ --cuda-graph-bs 2 8 16 24 32 36 40 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 58982` = `int(65536 * 0.9)` is the shared prefix portion. `--gsp-question-len 6553` = `int(65536 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name generated-shared-prefix \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --gsp-num-groups 1 \ --gsp-prompts-per-group 40 \ --gsp-system-prompt-len 58982 \ --gsp-question-len 6553 \ --gsp-output-len 1024 \ --max-concurrency 40 \ --num-prompts 40 \ --request-rate inf ``` ### Qwen3.6-35B-A3B 2P IN984K OUT1K **Model**: Qwen3.6-35B-A3B **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: BF16 **Dataset**: 984k+1k **TPOT**: 40.91ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --tp-size 4 \ --nnodes 1 \ --attention-backend ascend \ --device npu \ --chunked-prefill-size 131072 \ --max-prefill-tokens 984000 \ --disable-radix-cache \ --trust-remote-code \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.68 \ --cuda-graph-bs 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --context-length 1010000 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --dataset-path /path/to/dataset \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 984000 \ --random-output-len 1000 \ --num-prompts 1 \ --random-range-ratio 1 ``` # Qwen3-8B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b This page focuses on optimal configuration and benchmark results for Qwen3-8B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-8B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | -------- | ------------- | ----- | ----------- | --------- | ------- | ------------ | ----------------------------------------------------------- | | Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | | Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | -------- | ------------- | ----- | ----------- | --------- | ---- | ------------ | ------------------------------------------------------------ | | Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | ## Optimal Configuration ### Qwen3-8B W8A8 1P IN3K5 OUT1K5 37ms **Model**: Qwen3-8B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 37ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 70 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size 16384 \ --tp-size 1 \ --mem-fraction-static 0.85 \ --cuda-graph-bs 8 12 24 36 48 51 55 60 63 64 66 68 70 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 64 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 256 \ --random-range-ratio 1 ``` ### Qwen3-8B W8A8 1P IN3K5 OUT1K5 5ms **Model**: Qwen3-8B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 5ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 1 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ --cuda-graph-bs 1 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 4 \ --random-range-ratio 1 ``` ### Qwen3-8B W8A8 1P IN6K OUT1K5 BS16 **Model**: Qwen3-8B **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 11.79ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --nnodes 1 \ --node-rank 0 \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --max-running-requests 16 \ --max-prefill-tokens 16384 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ --cuda-graph-bs 1 5 15 16 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 4 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 5 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --num-prompts 16 \ --random-range-ratio 1 ``` # Qwen3-Next-80B-A3B-Instruct Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct This page focuses on optimal configuration and benchmark results for Qwen3-Next-80B-A3B-Instruct on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-Next-80B-A3B-Instruct Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct). On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------------------- | ------------- | ----- | ----------- | --------- | ------- | ------------ | ------------------------------------------------------------------------------- | | Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) | | Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --------------------------- | ------------- | ----- | ----------- | --------- | ---- | ------------ | ------------------------------------------------------------------------------- | | Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration ### Qwen3-Next-80B-A3B-Instruct W8A8 1P IN3K5 OUT1K5 20ms **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Atlas 800I A3 **Cards**: 1 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 20ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_WARMUP_TIMEOUT=3600 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export ZBCCL_ENABLE_GRAPH=1 export ZBCCL_LOCAL_MEM_SIZE=60416 export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 2 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --max-prefill-tokens 28672 \ --context-length 26384 \ --max-total-tokens 122304 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 2 \ --cuda-graph-bs 2 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 1 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 1 \ --random-range-ratio 1 ``` ### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN3K5 OUT1K5 50ms **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 3.5k+1.5k **TPOT**: 50ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=64 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_WARMUP_TIMEOUT=3600 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 export STREAMS_PER_DEVICE=32 export ZBAL_ENABLE_GRAPH=1 export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.75 \ --disable-radix-cache \ --max-prefill-tokens 14080 \ --context-length 26384 \ --chunked-prefill-size -1 \ --max-running-requests 300 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode auto \ --cuda-graph-bs 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \ --reasoning-parser qwen3 \ --tool-call-parser qwen ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 300 \ --random-input-len 3500 \ --random-output-len 1500 \ --num-prompts 300 \ --random-range-ratio 1 ``` ### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN6K OUT1K5 BS16 **Model**: Qwen3-Next-80B-A3B-Instruct **Hardware**: Atlas 800I A3 **Cards**: 2 **Deploy Mode**: PD Mixed **Quantization**: W8A8 INT8 **Dataset**: 6k+1.5k **TPOT**: 15.62ms #### Model Deployment ```bash Command theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DRAFT_MODEL_PATH: path to the draft model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ MODEL_PATH=/path/to/model-weights DRAFT_MODEL_PATH=/path/to/draft-model-weights echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export ASCEND_USE_FIA=1 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export FORCE_DRAFT_MODEL_NON_QUANT=1 export GLOO_SOCKET_IFNAME= export HCCL_BUFFSIZE=2000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_WARMUP_TIMEOUT=3600 export STREAMS_PER_DEVICE=32 export TASK_QUEUE_ENABLE=1 export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 export ZBCCL_ENABLE_GRAPH=1 export ZBCCL_LOCAL_MEM_SIZE=60416 export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --host 127.0.0.1 --port 6688 \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --page-size 128 \ --tp-size 4 \ --watchdog-timeout 9000 \ --mem-fraction-static 0.85 \ --disable-radix-cache \ --max-prefill-tokens 28672 \ --context-length 81920 \ --max-total-tokens 122304 \ --dp-size 2 \ --enable-dp-attention \ --enable-dp-lm-head \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 16 \ --cuda-graph-bs 2 4 8 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` #### Benchmark We tested it based on the `RANDOM` dataset. ```bash Command theme={null} python -m sglang.bench_serving \ --dataset-name random \ --backend sglang \ --host 127.0.0.1 \ --port 6688 \ --max-concurrency 16 \ --random-input-len 6144 \ --random-output-len 1500 \ --num-prompts 16 \ --random-range-ratio 1 ``` # DeepSeek-R1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1 ## Introduction DeepSeek-R1 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 671B total parameters with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in multi-token prediction (MTP) for speculative decoding. The model excels at reasoning, math, and code tasks through reinforcement learning-based training. This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-R1) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | MLAPO | `export SGLANG_NPU_USE_MLAPO=1` | | Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` | | NZ Weight Format | `export SGLANG_USE_FIA_NZ=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version, 376GB) * [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB) The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#single-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [DeepSeek-R1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # DeepSeek-V3.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2 ## Introduction DeepSeek-V3.2 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 685B total parameters with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in multi-token prediction (MTP) for speculative decoding. DeepSeek-V3.2 introduces the DeepSeek Sparse Attention (DSA) mechanism, a fine-grained sparse attention mechanism powered by a lightning indexer, enabling significant efficiency improvements in long-context scenarios. This document demonstrates the deployment of DeepSeek-V3.2 on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V3.2) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding, DSA context parallel), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 32` | | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 32 \`
`--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | Context Parallelism | `--enable-nsa-prefill-context-parallel \`
`--nsa-prefill-cp-mode in-seq-split \`
`--attn-cp-size 32` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version, 694.47GB) The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [DeepSeek-V3.2 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [DeepSeek-V3.2 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # GLM-5.1 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1 ## Introduction GLM-5.1 is a Mixture-of-Experts (MoE) large language model developed by Z.ai, featuring 744B total parameters with 40B active parameters. It uses 256 routed experts (top-8) plus one shared expert, with Multi-head Latent Attention (MLA) and DeepSeek Sparse Attention (DSA), and a built-in multi-token prediction (MTP) head for speculative decoding. The model features built-in bilingual (Chinese-English) capabilities with a unified pre-training framework, excelling at reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct mode (direct response), with a native context window of approximately 200k tokens. This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (GLM-5.1) is fully supported in this version. To use the latest features (e.g., speculative decoding, multi-node deployment), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Context Parallelism | `--enable-nsa-prefill-context-parallel \`
`--nsa-prefill-cp-mode in-seq-split \`
`--attn-cp-size 4` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16, 1.51TB) * [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version, 420.17GB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD mixed deployment Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [GLM-5.1 Best Practice — Multi-node PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [GLM-5.1 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [GLM-5.1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # GLM-5.2 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2 ## Introduction GLM-5.2 is a large language model in the GLM (General Language Model) series, jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. GLM-5.2 adopts the DeepSeek-V3/V3.2 architecture, including DeepSeek Sparse Attention (DSA) and multi-token prediction (MTP), and supports high-throughput inference with SGLang on Ascend NPUs. This document demonstrates the deployment of GLM-5.2 on Ascend NPUs using SGLang, including single-node deployment, multi-node deployment, prefill-decode disaggregation, feature configuration, and performance optimization. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [GLM-5.2](https://www.modelscope.cn/models/ZhipuAI/GLM-5.2) (BF16, 1.51TB) * [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP, 774.08GB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively. We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node deployment Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node. Run the following script to execute online inference. ```shell theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # ============================================================ # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 # MTP OVERLAP export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export HCCL_BUFFSIZE=1000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo # DEEPEP export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 MODEL_PATH=/path/to/model-weights python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --attention-backend ascend \ --device npu \ --tp-size 16 --nnodes 1 --node-rank 0 \ --chunked-prefill-size 16384 --max-prefill-tokens 280000 \ --trust-remote-code \ --host 127.0.0.1 \ --mem-fraction-static 0.7 \ --port 8000 \ --served-model-name glm-5 \ --cuda-graph-bs 16 \ --quantization modelslim \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --moe-a2a-backend deepep --deepep-mode auto ``` ### Multi-node deployment Quantized model `GLM-5.2-w8a8` can be deployed on two Atlas 800I A3 nodes. Modify the IP addresses of the two nodes, then run the same script on both nodes. ```shell theme={null} # ============================================================ # Before running, update the following variables: # IPS: IP addresses of each node in the cluster # IP_MASTER: rank 0 node IP address with port # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export STREAMS_PER_DEVICE=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 # MTP OVERLAP export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export HCCL_BUFFSIZE=1000 export HCCL_OP_EXPANSION_MODE=AIV # Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # DEEPEP export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 IPS=('' '') IP_MASTER="${IPS[0]}:5000" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` for i in "${!IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]]; then echo "${IPS[$i]}" python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ --attention-backend ascend \ --device npu \ --tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $IP_MASTER \ --chunked-prefill-size 16384 --max-prefill-tokens 131072 \ --trust-remote-code \ --host 127.0.0.1 \ --mem-fraction-static 0.8 \ --port 8000 \ --served-model-name glm-5 \ --cuda-graph-max-bs 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --disable-radix-cache NODE_RANK=$i break fi done ``` ### Prefill-decode disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. ```shell theme={null} # ============================================================ # Before running, update the following variables: # ASCEND_MF_STORE_URL: prefill master IP address with port # P_IP: prefill node IP address # D_IP: decode node IP address # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 export SGLANG_SET_CPU_AFFINITY=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # pd transfer, prefill master IP export ASCEND_MF_STORE_URL="tcp://:24707" export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 P_IP=('') D_IP=('') MODEL_PATH=/path/to/model-weights export TRANSFORMERS_VERBOSITY=error LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` echo "${LOCAL_HOST1}" echo "${LOCAL_HOST2}" # prefill for i in "${!P_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; then echo "${P_IP[$i]}" export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export TASK_QUEUE_ENABLE=2 export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # prefill node python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ --port 8000 --disaggregation-bootstrap-port 8998 --trust-remote-code --nnodes 1 --node-rank $i \ --tp-size 16 --mem-fraction-static 0.8 --attention-backend ascend --device npu --quantization modelslim \ --disaggregation-transfer-backend ascend --max-running-requests 64 \ --served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ --dp-size 4 --enable-dp-attention \ --load-balance-method round_robin \ --enable-dp-lm-head --moe-dense-tp 1 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 # cp #--enable-nsa-prefill-context-parallel \ #--nsa-prefill-cp-mode in-seq-split \ #--attn-cp-size 4 \ NODE_RANK=$i break fi done # decode for i in "${!D_IP[@]}"; do if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; then echo "${D_IP[$i]}" export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export HCCL_BUFFSIZE=650 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export TASK_QUEUE_ENABLE=0 export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= export SGLANG_NPU_USE_MULTI_STREAM=1 python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ --port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \ --mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \ --served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \ --cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ --tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 NODE_RANK=$i break fi done ``` Launch the router after the prefill and decode services are ready. ```shell theme={null} # ============================================================ # Before running, update the following variables: # P_MASTER_IP: prefill master IP address # D_MASTER_IP: decode master IP address # ROUTER_HOST_IP: router node IP address # ============================================================ P_MASTER_IP="" D_MASTER_IP="" ROUTER_HOST_IP="" python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy round_robin \ --prefill http://${P_MASTER_IP}:8000 8998 \ --decode http://${D_MASTER_IP}:8003 \ --host ${ROUTER_HOST_IP} \ --port 6688 ``` ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 8000) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference GLM-5.2 has no standalone best practice page yet, as tuning parameters are not finalized. Instead, the [Online service deployment](#online-service-deployment) section above provides ready-to-use scripts for single-node, multi-node, and PD disaggregation, each embedding the recommended feature combinations and tuning parameters (e.g., DeepEP mode, speculative decoding, overlap schedule). For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Hy3 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3 ## Introduction Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. It features 192 experts with top-8 activation per token, GQA attention (64 heads, 8 KV heads, head dim 128), and a 256K context length. The model supports built-in reasoning via `reasoning_effort`, tool calling, and multi-token prediction (MTP) for speculative decoding. Hy3 significantly outperforms similar-size models and rivals flagship open-source models with 2–5× the parameters, with notable improvements in agentic workflows, long-context tasks, and production reliability. This document demonstrates the deployment of Hy3 on Ascend NPUs using SGLang, including single-node (Atlas 800I A3) and multi-node (Atlas 800I A2) PD mixed mode and speculative decoding. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs`; e.g. `--cuda-graph-bs 4 8 16 20 24 28 32 36 40` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought;
`"reasoning_effort": "low"` — short thinking chain;
`"reasoning_effort": "no_think"` — direct response, no thinking | | Tool Calling | `--tool-call-parser auto --reasoning-parser auto` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Hy3 has 295B total parameters. If you need to download model weights, check the model size before downloading to reserve enough space. * [Hy3](https://www.modelscope.cn/models/Tencent-Hunyuan/Hy3) (BF16, 597.60GB) The BF16 variant (597.60GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full Atlas 800I A3 node (8 cards × 2 dies) or two Atlas 800I A2 nodes (8 cards × 1 die each). It is recommended to download the model weights to a shared directory accessible from within the container. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=64g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=64g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. * `--shm-size=64g` is recommended for 16-NPU deployments with large batch sizes. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode). Hy3 with 295B parameters requires all 16 logical NPUs on an Atlas 800I A3 server (8 physical NPUs × 2 logical cores each, 64 GB HBM per logical NPU). Set the following environment variables before launching the server: ```bash theme={null} # Performance tuning export SGLANG_SET_CPU_AFFINITY=1 echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 # CANN environment (adjust paths to match your installation) source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh # Ascend NPU settings export ASCEND_USE_FIA=1 export STREAMS_PER_DEVICE=32 export HCCL_BUFFSIZE=3000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo # SGLang settings export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` Launch the server: ```bash theme={null} MODEL_PATH="/path/to/Hy3" python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --attention-backend ascend \ --reasoning-parser auto \ --tool-call-parser auto \ --device npu \ --tp-size 16 \ --host 0.0.0.0 \ --port 9999 \ --mem-fraction-static 0.84 \ --dtype bfloat16 \ --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ --cuda-graph-bs 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 ``` ### Multi-node PD mixed deployment on Atlas 800I A2 Each Atlas 800I A2 node provides 8 NPUs (Ascend 910B, 64 GB HBM per NPU). Hy3 requires all 16 logical NPUs across **2 × Atlas 800I A2** nodes in PD mixed mode (prefill and decode are colocated on the same 2-node cluster). On each node, set the following environment variables before launching the server: ```bash theme={null} # ============================================================ # Before running, update the following variables: # HCCL_SOCKET_IFNAME: network interface name for HCCL (use `ifconfig` to find) # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ # Performance tuning export SGLANG_SET_CPU_AFFINITY=1 echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 # CANN environment (adjust paths to match your installation) source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh # Ascend NPU settings export ASCEND_USE_FIA=1 export STREAMS_PER_DEVICE=32 export HCCL_BUFFSIZE=3000 export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # SGLang settings export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` Launch the server on both nodes by running the same script on each node. `NODE_IPS[0]` is the master node and must be reachable from the other node: ```bash theme={null} # ============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory (shared) # NODE_IPS: IP addresses of the two nodes, e.g. ('10.0.0.1' '10.0.0.2') # ============================================================ MODEL_PATH="/path/to/Hy3" NODE_IPS=('' '') for i in "${!NODE_IPS[@]}"; do if [[ "$(hostname -I | awk '{print $1}')" == "${NODE_IPS[$i]}" ]]; then python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --attention-backend ascend \ --reasoning-parser auto \ --tool-call-parser auto \ --device npu \ --tp-size 16 \ --nnodes 2 \ --dist-init-addr ${NODE_IPS[0]}:5000 \ --node-rank $i \ --host 0.0.0.0 \ --port 9999 \ --mem-fraction-static 0.84 \ --dtype bfloat16 \ --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ --cuda-graph-bs 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 break fi done ``` * `HCCL_SOCKET_IFNAME` and `GLOO_SOCKET_IFNAME` must be set to the cluster network interface on both nodes. Do **not** use `lo` for multi-node deployments. * Model weights must be accessible from both nodes, e.g. mounted from a shared directory. * Send requests only to the master node (`NODE_IPS[0]:9999`), not to the worker node. ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 9999) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Hy3", "messages": [{"role": "user", "content": "What is the capital of France?"}], "max_tokens": 64, "temperature": 0.9, "extra_body": {"chat_template_kwargs": {"reasoning_effort": "no_think"}} }' ``` Expected result: an HTTP 200 response with `"Paris"` in the content field. To enable chain-of-thought reasoning for math or coding tasks, set `reasoning_effort` to `"high"`: ```shell theme={null} curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Hy3", "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 22, what is x?"}], "max_tokens": 8192, "temperature": 0.9, "extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}} }' ``` The response will contain a `reasoning_content` field with the thinking process and a `content` field with the final answer. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference Hy3 has no standalone best practice page yet. The [Single-node online deployment](#single-node-online-deployment) and [Multi-node PD mixed deployment on Atlas 800I A2](#multi-node-pd-mixed-deployment-on-atlas-800i-a2) sections above provide the ready-to-use scripts for the supported PD mixed cases. For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ### Reasoning mode selection Hy3 supports three reasoning modes via `chat_template_kwargs`: | `reasoning_effort` | Behavior | Recommended for | | ------------------ | --------------------------------- | ------------------------------- | | `no_think` | No chain-of-thought, direct reply | Simple Q\&A, low-latency tasks | | `low` | Short thinking chain | Moderate reasoning tasks | | `high` | Full chain-of-thought | Math, coding, complex reasoning | Use `no_think` for interactive or latency-sensitive applications to reduce output token count significantly. ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). This section only covers model-specific issues. # Kimi-K2.6 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6 ## Introduction Kimi-K2.6 is an open-source, native multimodal agentic model developed by Moonshot AI, built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It is a Mixture-of-Experts (MoE) model featuring Multi-head Latent Attention (MLA) and MoE architecture, with 1T total parameters and 32B active parameters. The model seamlessly integrates vision and language understanding with advanced agentic capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms. This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Kimi-K2.6) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 4 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 5 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | MLAPO | `export SGLANG_NPU_USE_MLAPO=1` | | Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB) * [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB) * [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi K2.6 Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#single-node-pd-mixed). ### Multi-node online deployment Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi-K2.6 Best Practice — Multi-node On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Kimi-K2.6 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Kimi-K2.6-w4a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Kimi-K2.6 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # MiMo-V2-Flash Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash ## Introduction MiMo-V2-Flash is a Mixture-of-Experts (MoE) large language model developed by Xiaomi. It employs advanced architecture with speculative decoding capabilities for accelerated inference. The model is optimized for high throughput and low latency scenarios through PD disaggregation deployment. This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD disaggregation mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (MiMo-V2-Flash) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 8` (prefill) or `--tp-size 16` (decode) | | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--enable-multi-layer-eagle` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` | | DP LM Head | `--enable-dp-lm-head` | | DP Attention | `--enable-dp-attention` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB) The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash#pd-disaggregation). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 9903) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [MiMo-V2-Flash Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # MiniMax-M2.5 Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5 ## Introduction MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and long-context tasks. This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (MiniMax-M2.5) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. * [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version, 230.82GB) * [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding) The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [MiniMax-M2.5 Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [MiniMax-M2.5 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-235B-A22B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b ## Introduction Qwen3-235B-A22B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 235B total parameters with 22B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-235B-A22B) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 16` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | | Context Parallelism | `--enable-prefill-context-parallel \`
`--attn-cp-size 2 \`
`--moe-dp-size 2` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16, 470.21GB) * [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version, 236.80GB) * [Qwen3-235B-A22B-Eagle3](https://www.modelscope.cn/models/nv-community/Qwen3-235B-A22B-Eagle3) (EAGLE3 draft model for speculative decoding) The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-235B-A22B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b#single-node-pd-mixed). ### Multi-node PD disaggregation deployment #### 256K long-sequence PD disaggregation on 2 x Atlas 800I A3 (without CP) This configuration uses PD disaggregation for **256K long-sequence inference** on 2 x Atlas 800I A3 with context parallel disabled. The following command is based on the **W8A8** quantized model. 1. Set the shared environment variables on both prefill and decode nodes: ```bash Shared environment theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # PREFILL_HOST_IP: prefill node IP address # NETWORK_IFACE: network interface name (use ifconfig to find) #============================================================ export ASCEND_USE_FIA=1 export SGLANG_SET_CPU_AFFINITY=1 export ASCEND_MF_STORE_URL="tcp://:12345" export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= MODEL_PATH=/path/to/model-weights ``` 2. Run on the **prefill node**: ```bash Prefill node theme={null} #============================================================ # Before running, update the following variable: # PREFILL_HOST_IP: prefill node IP address #============================================================ export ASCEND_LAUNCH_BLOCKING=1 export HCCL_BUFFSIZE=1500 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 export DEEPEP_NORMAL_LONG_SEQ_ROUND=128 export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --attention-backend ascend \ --disable-radix-cache \ --chunked-prefill-size -1 \ --skip-server-warmup \ --device npu \ --quantization modelslim \ --tp-size 16 \ --mem-fraction-static 0.45 \ --max-running-requests 1 \ --host \ --port 8000 \ --dist-init-addr :5000 \ --nnodes 1 \ --node-rank 0 \ --moe-a2a-backend deepep \ --deepep-mode normal ``` 3. Run on the **decode node**: ```bash Decode node theme={null} #============================================================ # Before running, update the following variable: # DECODE_HOST_IP: decode node IP address #============================================================ export HCCL_BUFFSIZE=4000 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 export DEEPEP_NORMAL_LONG_SEQ_ROUND=16 python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --attention-backend ascend \ --mem-fraction-static 0.8 \ --disable-cuda-graph \ --device npu \ --quantization modelslim \ --disable-radix-cache \ --chunked-prefill-size 8192 \ --skip-server-warmup \ --tp-size 16 \ --max-running-requests 1 \ --host \ --port 8232 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ --disable-overlap-schedule ``` 4. Launch the SGLang Router (on any reachable node): ```bash Router theme={null} #============================================================ # Before running, update the following variables: # PREFILL_HOST_IP: prefill node IP address # DECODE_HOST_IP: decode node IP address # ROUTER_HOST_IP: router node IP address #============================================================ python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ --prefill http://:8000 8995 \ --decode http://:8232 \ --host \ --port 6689 \ --prometheus-port 29010 ``` #### Prefill Context Parallel (PCP) on 2 x Atlas 800I A3 This configuration enables **Prefill Context Parallel** (`--enable-prefill-context-parallel`) to split the context across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences. PD disaggregation is required. The following command is based on the **W8A8** quantized model. **Constraints:** * Prefill side must set `--max-running-requests 1` (PCP only supports batch\_size=1) * `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs 1. Run on the **prefill node**: ```bash Prefill node theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # PREFILL_HOST_IP: prefill node IP address #============================================================ export SGLANG_SET_CPU_AFFINITY=1 export ASCEND_MF_STORE_URL="tcp://:23456" export ASCEND_USE_FIA=True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --trust-remote-code \ --disaggregation-mode prefill \ --disaggregation-transfer-backend ascend \ --disaggregation-bootstrap-port 8995 \ --quantization modelslim \ --attention-backend ascend \ --skip-server-warmup \ --mem-fraction-static 0.7 \ --chunked-prefill-size 32768 \ --device npu \ --base-gpu-id 0 \ --tp-size 16 \ --enable-prefill-context-parallel \ --attn-cp-size 2 \ --moe-dp-size 2 \ --max-running-requests 1 \ --host \ --port 8000 \ --nnodes 1 \ --node-rank 0 \ --dist-init-addr :6688 ``` Key parameters for PCP: | Parameter | Value | Description | | ----------------------------------- | ----- | --------------------------------------------------------------------- | | `--enable-prefill-context-parallel` | flag | Enable PCP feature | | `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) | | `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` | | `--max-running-requests` | 1 | Required by PCP (batch\_size=1 constraint) | 2. Run on the **decode node**: ```bash Decode node theme={null} #============================================================ # Before running, update the following variables: # MODEL_PATH: path to the model weights directory # DECODE_HOST_IP: decode node IP address # PREFILL_HOST_IP: prefill node IP address (for ASCEND_MF_STORE_URL) #============================================================ export ASCEND_MF_STORE_URL="tcp://:23456" export ASCEND_USE_FIA=True python3 -m sglang.launch_server \ --model-path ${MODEL_PATH} \ --trust-remote-code \ --disaggregation-mode decode \ --disaggregation-transfer-backend ascend \ --quantization modelslim \ --attention-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ --mem-fraction-static 0.7 \ --chunked-prefill-size 32768 \ --skip-server-warmup \ --device npu \ --base-gpu-id 0 \ --tp-size 8 \ --max-running-requests 32 \ --host \ --port 8001 \ --nnodes 1 \ --node-rank 0 \ --dist-init-addr :6688 ``` `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the prefill node IP). `ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer. PCP is a prefill-only feature; the decode side needs no CP-related flags. ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6689) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-235B-A22B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-30B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b ## Introduction Qwen3-30B-A3B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 30B total parameters with 3B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-30B-A3B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-30B-A3B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 2` | | Data Parallelism | `--dp-size 2` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, 61.08GB, recommended) * [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version, 31.29GB) * [Qwen3-a3B\_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding) Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-30B-A3B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-30B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-32B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b ## Introduction Qwen3-32B is a dense model in the Qwen3 series developed by Alibaba, featuring 32B parameters with Grouped-Query Attention (GQA) and up to 128k (131k with YaRN) context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and supports both standard and thinking/reasoning modes. This document demonstrates the deployment of Qwen3-32B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-32B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 4` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16, 65.54GB) * [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version, 42.77GB) * [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding) The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. The W8A8 variant (42.77GB) fits within a single die. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-32B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-32B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.5-397B-A17B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b ## Introduction Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, featuring a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated). It employs hybrid attention with Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer, and MoE routing with Top-10 active out of 512 routed experts plus a dedicated shared expert. The model supports multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token prediction (MTP) for speculative decoding. This document demonstrates the deployment of Qwen3.5-397B-A17B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.5-397B-A17B) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB) The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.5-397B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.5-397B-A17B-w4a8-mtp", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.5-397B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.6-27B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b ## Introduction Qwen3.6-27B is a dense model in the Qwen3.6 series developed by Alibaba, featuring 27B parameters with a hybrid architecture combining Gated Delta Networks (linear, O(n) complexity) with full attention every 4th layer. It supports multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token prediction (MTP) for speculative decoding. The model delivers strong performance in instruction following, reasoning, text comprehension, and tool usage. This document demonstrates the deployment of Qwen3.6-27B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-27B) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB) The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.6-27B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.6-27B-w8a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.6-27B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.6-35B-A3B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b ## Introduction Qwen3.6-35B-A3B is a Mixture-of-Experts model in the Qwen3.6 series developed by Alibaba, featuring 35B total parameters with 3B active parameters per token. It uses a hybrid architecture that combines Gated DeltaNet (linear, O(n) complexity) with full attention applied every Nth layer, enabling efficient long-context modeling. The model supports multimodal inputs (text, image, video) with long context lengths, and includes built-in multi-token prediction (NEXTN) for speculative decoding. It delivers strong performance in instruction following, reasoning, text comprehension, and tool usage. This document demonstrates the deployment of Qwen3.6-35B-A3B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-35B-A3B) is fully supported in this version. To use the latest features (e.g., speculative decoding / NEXTN, multimodal), use **v0.5.16 or a later version**. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 2` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB) The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.6-35B-A3B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.6-35B-A3B-w8a8", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.6-35B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3.8-Max Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max ## Introduction Qwen3.8-Max (published as Qwen3.8-2.4T-A95B) is a Mixture-of-Experts (MoE) model with 2.4 trillion total parameters and 95B activated per token. It uses a 92-layer hybrid architecture that combines Gated Delta Network (GDN) linear-attention with full-attention layers. The model has a native context length of 262,144 tokens (extensible to over 1 million) and built-in multi-token prediction (MTP) weights for speculative decoding. This document demonstrates the deployment of Qwen3.8-Max on Ascend NPUs using SGLang, including multi-node PD mixed mode, feature configuration, and performance optimization. Qwen3.8-Max is newly released. This document is validated and written based on the **SGLang main branch (daily build)**. It is recommended to use the latest daily build Docker image, or build SGLang from the main branch source. ## Supported features | Feature | Example usage | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 64` | | Data Parallelism | `--enable-dp-attention \`
`--dp-size 4` | | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 16` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen/Qwen3.8-2.4T-A95B](https://www.modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B) (BF16, approximately 4.8TB) * You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize a W4A8 variant from the BF16 checkpoint. We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. The validated configuration in this tutorial deploys the W4A8 variant on 4 Atlas 800I A3 nodes (`--tp-size 64`, 64 dies in total). The BF16 checkpoint alone weighs approximately 4.8TB and requires additional nodes. For the hardware specifications (memory per die, dies per card), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. The following command is based on the daily build tag, which contains the latest SGLang main branch changes. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:main-cann9.0.0-a3 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Multi-node online deployment Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. The validated configuration deploys the W4A8 quantized checkpoint on 4 nodes with `--tp-size 64` (16 dies per node), DP attention (`--dp-size 4`), and DeepEP in `auto` mode. Modify the IP addresses of the four nodes, then run the same script on all four nodes. Each node determines its own rank by matching the local IP address against `IPS`. ```shell theme={null} # ============================================================ # Before running, update the following variables: # IPS: IP addresses of each node in the cluster # MODEL_PATH: path to the model weights directory # HCCL_SOCKET_IFNAME: network interface name for HCCL # GLOO_SOCKET_IFNAME: network interface name for Gloo # ============================================================ # high performance cpu echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=10 sysctl -w kernel.numa_balancing=0 sysctl -w kernel.sched_migration_cost_ns=50000 # bind cpu export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS=1 unset https_proxy unset http_proxy unset HTTPS_PROXY unset HTTP_PROXY unset ASCEND_LAUNCH_BLOCKING # cann source /usr/local/Ascend/ascend-toolkit/set_env.sh source /usr/local/Ascend/nnal/atb/set_env.sh export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export STREAMS_PER_DEVICE=32 # deepep export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export HCCL_BUFFSIZE=2300 export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512 export HCCL_OP_EXPANSION_MODE=AIV # Run command ifconfig on each node, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= IPS=('' '' '' '') IP_MASTER="${IPS[0]}:5000" MODEL_PATH=/path/to/model-weights LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` for i in "${!IPS[@]}"; do if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]]; then echo "${IPS[$i]}" # overlap schedule export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_RAGGED_VERIFY_MODE=static sglang serve \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --dist-init-addr $IP_MASTER --nnodes 4 --node-rank $i \ --model-path $MODEL_PATH \ --tokenizer-path $MODEL_PATH \ --trust-remote-code \ --attention-backend ascend \ --device npu \ --quantization modelslim \ --dtype bfloat16 \ --tp-size 64 \ --enable-dp-attention --dp-size 4 --enable-dp-lm-head \ --mem-fraction-static 0.8 \ --chunked-prefill-size 8192 \ --cuda-graph-bs 16 \ --disable-radix-cache \ --max-running-requests 64 \ --host 0.0.0.0 \ --port 30000 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --watchdog-timeout 9000 break fi done ``` ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 30000) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference Qwen3.8-Max has no standalone best practice page yet, as tuning parameters are not finalized. Instead, the [Online service deployment](#online-service-deployment) section above provides a ready-to-use script for multi-node deployment, which embeds the recommended feature combinations and tuning parameters (e.g., DP attention, DeepEP, overlap schedule). For the full catalog of optimization features and their parameter and compatibility details, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-8B Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b ## Introduction Qwen3-8B is a compact dense model in the Qwen3 series developed by Alibaba, featuring 8B parameters with Grouped-Query Attention (GQA) and up to 128k context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and is available in both standard and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-8B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-8B) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16, 16.40GB) * [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version, 11.27GB) * [Eagle3-Qwen3-8B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh) (EAGLE3 draft model for speculative decoding) We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 or A3. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-8B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-8B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Qwen3-Next-80B-A3B-Instruct Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct ## Introduction Qwen3-Next-80B-A3B-Instruct is a hybrid Gated DeltaNet-Transformer Mixture-of-Experts (MoE) model developed by Alibaba, featuring 80B total parameters with 3B active parameters. It combines Gated DeltaNet SSM layers with attention layers in a sparse MoE architecture, enabling efficient long-sequence modeling with linear complexity in the SSM layers. The model supports multi-token prediction (NEXTN) for speculative decoding, delivering strong performance in instruction following, reasoning, and text generation tasks. This document demonstrates the deployment of Qwen3-Next-80B-A3B-Instruct on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-Next-80B-A3B-Instruct) is fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later version. ## Supported features | Feature | Example usage | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 4` | | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--ep-size 4 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant \`
`--speculative-draft-model-path /path/to/draft-model-weights` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16, 162.68GB) — used as the EAGLE3 draft model * [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version, 84.90GB) We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput. It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. The BF16 weights serve as the EAGLE3 draft model (set `--speculative-draft-model-path` to the BF16 weight path). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3-Next-80B-A3B-Instruct Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3-Next-80B-A3B-Instruct Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq). # Parameter Tuning Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning This guide explains the role of each parameter used in SGLang deployments on Ascend NPU. It uses the [DeepSeek-V3.2 best practice configuration](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation) as the reference example. For a complete list of tested deployment configurations, see the [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page. Parameters in this guide fall into two categories: * **Required configurations** (marked with `[Required]`): These must be set correctly for the target deployment scenario (e.g., multi-node communication, PD disaggregation). Incorrect values will cause deployment failures or incorrect behavior. * **Performance optimizations**: These improve throughput, latency, or memory efficiency. The optimal values depend on your specific model, hardware, and workload and may require tuning. Where the optimal value is not obvious, tuning guidance is provided. ## System-Level Optimizations The following system-level tuning steps reduce OS interference and improve CPU scheduling determinism:
Command / Variable Purpose
`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor` Locks all CPU cores to the maximum frequency, eliminating DVFS-induced latency jitter during inference-critical paths.
`sysctl -w vm.swappiness=0` Minimizes kernel swapping of anonymous pages. Reduces the risk of page faults on NPU memory buffers pinned to host RAM.
`sysctl -w kernel.numa_balancing=0` Disables automatic NUMA page migration. Prevents the kernel from moving memory pages between NUMA nodes while inference is running, which would cause latency spikes.
`sysctl -w kernel.sched_migration_cost_ns=50000` Sets a minimum task migration cost, discouraging the scheduler from moving inference threads between CPU cores unnecessarily.
`SGLANG_SET_CPU_AFFINITY=1` Binds SGLang worker processes to specific CPU cores, avoiding cross-core migration overhead for high-frequency scheduling loops.
## Memory & Device Configuration
Variable / Argument Purpose Reference Value
`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True` Enables the expandable NPU memory allocator, allowing the memory pool to grow dynamically. This avoids out-of-memory errors when workloads have variable memory requirements and is essential for large models such as MoE architectures. `expandable_segments:True`
`STREAMS_PER_DEVICE=32` Sets the maximum number of parallel streams per NPU device. More streams allow better overlap between compute and communication operations. The default of `32` is sufficient for most deployments; increase only if profiling reveals stream contention in complex pipeline parallelism setups. `32`
`--mem-fraction-static` Controls the fraction of NPU memory allocated to model weights and the KV cache pool. Lower values leave headroom for intermediate activations; higher values maximize KV cache capacity for serving more concurrent requests. The optimal value depends on your model size, sequence length, and available NPU memory. Start conservatively and increase gradually while monitoring for out-of-memory errors. Prefill: `0.73`, Decode: `0.79`
## Communication Configuration
Variable Purpose Reference Value
`HCCL_BUFFSIZE` Sets the HCCL communication buffer size in MB. Larger buffers increase throughput for bulk transfers but consume more host memory. The optimal value depends on your communication pattern — larger values benefit prefill (bulk token transfers), while smaller values are sufficient for decode (small batches). Tune based on your expected token dispatch volume. Prefill: `1200`, Decode: `400`
`HCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` \[Required for multi-node] Specifies the network interface used for HCCL and GLOO distributed communication. Must be set to the high-bandwidth inter-node network interface (e.g., RDMA-capable NIC) for multi-node deployments. Without this, the framework may default to a low-bandwidth interface, severely degrading distributed communication performance. Not needed for single-node deployments. Set per-cluster
## MoE & Expert Parallelism
Variable / Argument Purpose Reference Value
`--moe-a2a-backend` Selects the all-to-all communication backend for MoE expert dispatch and combine. On Ascend NPU, the primary options are `deepep` (DeepEP) and `ascend_fuseep` (Ascend Fused EP). DeepEP is optimized for large-scale models with flexible prefill/decode dispatch paths; `ascend_fuseep` provides a general fused MoE dispatch path. `deepep`
`--deepep-mode` Selects the DeepEP operating mode. Available options: `normal` (optimized for high throughput, long sequences, and large token counts — suitable for prefill), `low_latency` (optimized for low latency, CUDA Graph compatible, small batches — suitable for decode), and `auto` (switches automatically based on the operation type). Use `auto` if unsure; use explicit modes when managing prefill/decode independently in PD disaggregation. Prefill: `normal`, Decode: `low_latency`
`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` Sets the maximum number of tokens that a single rank can dispatch in one DeepEP operation (hard upper limit: `1024`). Larger values accommodate more tokens per dispatch but increase buffer allocation overhead. For prefill, use a large value or unbounded (`0`) since many tokens are processed. For decode, set to match your expected tokens per iteration. Must satisfy: `max-running-requests * (1 + draft_tokens) <= this value`. Prefill: `0` (unbounded), Decode: `8`
`DEEP_NORMAL_MODE_USE_INT8_QUANT` When set to `1`, quantizes intermediate activations to INT8 in the DeepEP dispatch operator, reducing communication volume during MoE dispatch. This is most beneficial for large-scale multi-node prefill with many tokens. The trade-off is a small accuracy impact from quantization and additional compute for the quantize/dequantize operations. Prefill: `1`
`TASK_QUEUE_ENABLE` Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with NPU Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where NPU Graph compatibility is needed. Prefill: `2`, Decode: `0`
`--moe-dense-tp-size` Sets the tensor parallelism size for MoE dense (shared) MLP layers. When using DP attention, setting this to `1` avoids an unnecessary all-reduce across the DP group for the dense MLP layers, since each DP shard already has the full weight. In deployments without DP attention, set this to match your TP size. `1`
## Prefill Optimizations These arguments and environment variables are critical for tuning prefill performance:
Argument / Variable Purpose Reference Value
`--chunked-prefill-size` Sets the maximum number of tokens per prefill chunk. A positive value enables chunked prefill, which interleaves prefill and decode for better concurrency in mixed workloads. Set to `-1` to disable chunking and process each request in a single forward pass, which is preferred for dedicated prefill servers with long-context sequences. `-1`
`--max-prefill-tokens` Limits the total number of tokens the prefill server can process in one batch. The effective bound is `max(this value, model_max_context_length)`. Set this based on your target sequence length and available NPU memory to bound memory usage while maximizing throughput. Tune by increasing until you encounter out-of-memory errors. `68000`
`--max-running-requests` Limits the number of concurrent requests being processed. For prefill, a low value (e.g., `1`) dedicates more compute and memory to each request, achieving higher per-request throughput — ideal for dedicated prefill nodes processing long sequences. For general-purpose serving, use a higher value to support multi-request concurrency. `1`
`--disable-radix-cache` Disables prefix caching via RadixAttention. Set this flag when processing non-overlapping long sequences where prefix caching provides no benefit and only consumes memory. Leave unset (radix cache enabled) for chat/conversation workloads with shared system prompts. true
`--disable-cuda-graph` Disables CUDA Graph capture. CUDA Graphs reduce kernel launch overhead for small, predictable batch sizes, making them ideal for decode. For prefill with large and variable batch sizes, CUDA Graphs provide minimal benefit and can cause issues with dynamic shapes. Set this flag on prefill nodes; leave unset on decode nodes. true
`--enable-dsa-prefill-context-parallel` (DeepSeek V3.2 DSA-specific) Enables context parallelism for the long-sequence prefill phase of DeepSeek V3.2 with DSA (DeepSeek Sparse Attention). Distributes the sequence across CP ranks to parallelize the computationally expensive DSA prefill for ultra-long contexts. Enabled
`--dsa-prefill-cp-mode` (DeepSeek V3.2 DSA-specific) Controls how the long sequence is split across context parallel ranks: `in-seq-split` divides each sequence uniformly across CP ranks, optimal for single-request prefill. `round-robin-split` (code default) distributes tokens by index mod CP size, supporting multi-batch prefill. Only effective when `--enable-dsa-prefill-context-parallel` is enabled. `in-seq-split`
`--attn-cp-size` Specifies the context parallelism group size for attention computation. Larger values distribute the sequence across more ranks, reducing per-rank memory and compute at the cost of increased communication. For models with DSA, this controls the CP size for sparse attention prefill. Set to the number of available devices for maximum parallelization. `32`
## Decode Optimizations These arguments and environment variables are critical for tuning decode performance:
Argument / Variable Purpose Reference Value
`--dp-size` / `--data-parallel-size` Sets the data parallelism degree for the decode server. With DP attention enabled, attention layers are sharded across DP ranks while FFN/MoE layers use tensor parallelism. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides evenly into your total card count, with remaining cards used for TP/EP. `8`
`--ep` / `--ep-size` / `--expert-parallel-size` Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value. `32`
`--enable-dp-attention` Enables data parallelism for attention layers while keeping tensor parallelism for FFN/MoE layers. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck. Enabled
`--enable-dp-lm-head` Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput. Enabled
`--cuda-graph-max-bs-decode` Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving. `4`
`SGLANG_SCHEDULER_SKIP_ALL_GATHER=1` When DP attention is enabled, the scheduler normally performs an all-gather across DP ranks to determine the full set of ready requests. Setting this to `1` skips that operation, reducing decode scheduling latency. Only safe when load is balanced across DP ranks (e.g., via a round-robin load balancing policy). Disable if you observe uneven load distribution across DP ranks. `1`
## Speculative Decoding Speculative decoding reduces per-token latency by generating draft tokens that are then verified by the target model:
Argument / Variable Purpose Reference Value
`--speculative-algorithm` Selects the speculative decoding algorithm. `NEXTN` (aliased to `EAGLE`) uses the model's built-in MTP (Multi-Token Prediction) heads, requiring no separate draft model. `EAGLE3` uses an external draft model, which can achieve higher acceptance rates at the cost of additional memory. Other built-in options include `STANDALONE`, `NGRAM`, and `DFLASH`, plus any plugin-registered name via `SpeculativeAlgorithm.register`. Choose `NEXTN` for models with native MTP support (e.g., DeepSeek V3.2/R1); choose `EAGLE3` for models without MTP (e.g., Qwen MoE). `NEXTN`
`--speculative-num-steps` Number of speculative forward passes per iteration. More steps can increase the acceptance length and throughput but add latency. For prefill, use a small value (`1`) to minimize prefill latency impact. For decode, use a larger value (`2`–`4`) to maximize throughput. Tune based on your latency vs throughput requirements. Prefill: `1`, Decode: `3`
`--speculative-eagle-topk` Limits the number of draft tokens considered per position. Lower values reduce compute on unlikely tokens and are required for the overlap scheduler (enabled by default). Higher values may increase acceptance rates but add overhead. Start with `1` for the overlap scheduler; otherwise, `4`–`8` is typical. `1`
`--speculative-num-draft-tokens` Number of draft tokens generated per speculative step. Higher values increase potential acceptance length and throughput but add per-step computation. Balance against your latency budget — prefill typically uses fewer draft tokens (`2`) to minimize overhead; decode can use more (`4`) to maximize throughput. Prefill: `2`, Decode: `4`
`SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` Enables the overlap plan stream feature for EAGLE v2/v3 speculative decoding workers. This overlaps draft model computation with target model verification, effectively hiding draft latency. Enable when using EAGLE-based speculative decoding; not applicable for NEXTN. `1`
## Quantization
Argument Purpose Reference Value
`--quantization modelslim` Uses the Ascend ModelSlim quantization tool to load W4A4/W4A8/W8A8/W4A16 pre-quantized model weights. This reduces model weight footprint by approximately 50% (for w8) or 75% (for w4) compared to BF16, allowing larger models to fit in NPU memory with minimal accuracy degradation. The quantization method is auto-detected from the model's `quant_model_description.json` file. `modelslim`
## Throughput Configuration
Argument / Variable Purpose Reference Value
`--tokenizer-worker-num` Sets the number of parallel tokenizer worker processes. Increasing this allows concurrent tokenization of multiple input/output streams, preventing the tokenizer from becoming a bottleneck under high request concurrency. Set based on your CPU core count and expected request rate. `4`
`--load-balance-method` Selects the DP load balancing strategy. Available options: `auto` (default, automatically selects the best strategy), `round_robin` (assigns requests to DP ranks in rotation for even distribution), `total_tokens` (balances by token load), `total_requests` (balances by request count), `follow_bootstrap_room` (follows the bootstrap room assignment). Start with `round_robin` for simple even distribution; use `total_tokens` if your requests have highly variable lengths. `round_robin`
`ASCEND_MF_STORE_URL` \[Required for PD disaggregation] Sets the MemFabric config store address for PD disaggregation. This is the prefill primary node's IP with an arbitrary port, used by the decode nodes to discover and connect to the MemFabric-based KV cache transfer service. Omit this for non-disaggregated deployments. Prefill IP with port
## Additional Ascend NPU-Specific Parameters The following environment variables are used in other best practice configurations and may be applicable depending on your model and deployment:
Variable Purpose Typical Usage
`HCCL_OP_EXPANSION_MODE=AIV` Configures the HCCL communication algorithm scheduling to use AIV (Ascend Intelligent Vision) expansion mode, which can improve communication efficiency for certain collective operations. Used in Qwen MoE and R1 non-DSA configurations
`SGLANG_NPU_USE_MLAPO=1` (DeepSeek MLA-specific) Adopts the `MLAPO` fusion operator in the MLA (Multi-Head Latent Attention) preprocessing stage for DeepSeek models with MLA architecture. Used with DeepSeek R1
`SGLANG_USE_FIA_NZ=1` (DeepSeek MLA-specific) Reshapes the KV Cache into FIA NZ format for improved memory access efficiency. Must be used together with `SGLANG_NPU_USE_MLAPO=1`. Used with DeepSeek R1
`SGLANG_NPU_USE_MULTI_STREAM=1` (DeepSeek MoE-specific) Enables dual-stream computation for shared experts and routing experts in DeepSeek MoE models, allowing the two expert types to execute concurrently on separate streams. Used with DeepSeek R1
`SGLANG_USE_AG_AFTER_QLORA=1` Delays the all-gather operation until after Q-LoRA processing. This reduces communication overhead by performing Q-LoRA projection before the all-gather, requiring fewer bytes to be transferred. Used with DeepSeek V3.2/R1 prefill
## Feature Compatibility The table shows feature-by-feature compatibility of SGLang capabilities on Ascend NPU. Features that have performance impact on Ascend are included. Each cell at the intersection of two features indicates whether they can be used together. The symbols used are defined as follows: * 🟢 = Full compatibility * 🟠 = Partial compatibility * ❌ = No compatibility * ❔ = Unknown or TBD
Feature Tensor Parallelism Data Parallelism Expert Parallelism Context Parallelism PD Disaggregation Quantization Chunked Prefill NPU Graph Speculative Decoding PrefixCache Overlap Schedule DP LM Head MLAPO Multistream MoE EPLB NZ Weight Format
Tensor Parallelism 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Data Parallelism 🟢 🟢 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Expert Parallelism 🟢 🟢 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢
Context Parallelism 🟢 🟠 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟠 ❔ 🟢 🟢 🟢 🟢
PD Disaggregation 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟠 🟢 🟠 🟠 🟢 🟠 🟢
Quantization 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢
Chunked Prefill 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟢 🟠 🟠 🟠 🟠 🟠 🟢
NPU Graph 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟠 🟢 🟠 🟠 🟠 🟠 🟠 🟠
Speculative Decoding 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟠 🟢 🟢 🟢 🟢
PrefixCache 🟢 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
Overlap Schedule 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟢
DP LM Head 🟢 🟢 🟠 ❔ 🟠 🟠 🟠 🟠 🟠 🟢 🟢 🟢 🟢 🟠 🟢
MLAPO 🟢 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 ❔ 🟢
Multistream MoE 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟠 🟢 🟢 🟢 🟢 🟢 ❔ 🟢
EPLB 🟢 🟢 🟢 🟢 🟠 🟢 🟠 🟠 🟢 🟢 🟢 🟠 ❔ ❔ 🟠
NZ Weight Format 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟠 🟢 🟢 🟢 🟢 🟢 🟢 🟠
## Feature descriptions ### Tensor Parallelism (`--tp-size`) Splits model weights across multiple NPU devices so that large models can be loaded and run cooperatively. Each device holds a shard of every weight tensor and communicates via HCCL all-reduce. This is the primary mechanism for deploying models that exceed single-device memory. ### Data Parallelism (`--dp-size`, `--enable-dp-attention`) Replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. With --enable-dp-attention, the attention layers are replicated across DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed. ### Expert Parallelism (`--ep-size`) Each device holds a subset of experts and routes tokens via all-to-all communication based on the model's gating network. Supports load balancing methods including default round-robin expert placement and dynamic Expert Parallelism Load Balancing (EPLB). ### Context Parallelism (`--attn-cp-size`) Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128k tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`. ### PD Disaggregation (`--disaggregation-mode`) Separates the prefill (P) and decode (D) phases onto different device groups, allowing each phase to be independently optimized for its compute and memory characteristics. Prefill nodes handle long input processing at high throughput, while decode nodes focus on low-latency token generation. Uses the Ascend MemFabric transfer backend (`--disaggregation-transfer-backend ascend`). ### Quantization (`--quantization`) Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput. On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static). Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors) depends on the availability of specific Ascend custom kernels in your environment; see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/quantization) for details and compatibility matrices. ### Chunked Prefill (`--chunked-prefill-size`) Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity. ### NPU Graph (`--cuda-graph-bs`) Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes. `--enable-torch-compile` is incompatible with NPU Graph. When torch.compile is enabled, NPU Graph must be disabled via `--disable-cuda-graph`. ### Speculative Decoding (`--speculative-algorithm`) Reduces per-token latency by predicting multiple future tokens in a single forward pass, then verifying them against the model. Ascend supports `NEXTN` (for DeepSeek models, using the model's own hidden states) and `EAGLE3` (for Qwen MoE models, using a separate draft model). Controlled by `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. On Ascend, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` enables the optimized overlap scheduler for speculative decoding. ### PrefixCache (`--disable-radix-cache`) Reuses KV cache across requests that share common prompt prefixes (Radix Cache), reducing repeated prefill computation and lowering time-to-first-token. Enabled by default; disable with `--disable-radix-cache` when prefix reuse is not expected (e.g., dedicated prefill nodes in PD disaggregation, or random-input benchmarks). ### Overlap Schedule On Ascend, this is primarily controlled via environment variable SGLANG\_ENABLE\_OVERLAP\_PLAN\_STREAM=1, which enables the optimized overlap scheduler for speculative decoding scenarios. Most effective during decode with speculative decoding and DP attention, where expert dispatch/reduce communication can be overlapped with the next batch's computation. ### DP LM Head (`--enable-dp-lm-head`) Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--enable-dp-attention`. ### MLAPO (`SGLANG_NPU_USE_MLAPO=1`) Enables the MLAPO (MLA Pre-Processing Optimization) fusion operator for DeepSeek-style Multi-head Latent Attention models, replacing multiple separate kernels with a single fused kernel for Q/K/V projection and absorption. Reduces prefill latency and memory bandwidth usage. Used in most DeepSeek best-practice configurations. ### Multistream MoE (`SGLANG_NPU_USE_MULTI_STREAM=1`) Enables dual-stream parallel execution for MoE layers, where shared experts and routing experts run concurrently on separate NPU streams. This overlaps shared-expert computation with routed-expert dispatch, improving MoE layer throughput. ### EPLB (`--enable-eplb`) Enables Expert Parallelism Load Balancing to dynamically redistribute experts across devices based on workload, ensuring even expert utilization and preventing hotspots. Uses the `deepseek` balancing algorithm. Compatible with the `deepep` MoE A2A backend, but not with `ascend_fuseep`. ### NZ Weight Format (`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`) Casts model weight tensors to the Ascend NPU ACL FRACTAL\_NZ format (format 29) for improved memory access efficiency on Da Vinci AI cores. Enabled by default globally; disable with `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT=1` when the format conflicts with other features (e.g., EPLB). Applied to linear weights, MoE weights, quantized weights, MLA QKV projections, and KV Cache (via `PA_NZ` / `nzcache` modes for FIA). Related env var: `SGLANG_USE_FIA_NZ=1` (requires `SGLANG_NPU_USE_MLAPO=1`). ## See Also * [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — Complete deployment configurations and benchmark results for all supported models * [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables) — Reference for all Ascend NPU-related environment variables * [DeepSeek V3.2 Guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) — Detailed usage guide for DeepSeek V3.2 deployment * [Expert Parallelism](/docs/advanced_features/expert_parallelism) — DeepEP configuration and tuning guide # Performance Profiling Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/profiling During inference serving, it is sometimes necessary to monitor the internal execution flow of the serving framework to identify performance issues. By collecting start/end timestamps of key flows, identifying critical functions or iterations, recording key events, and gathering relevant information, you can quickly locate performance bottlenecks. This guide walks you through the complete workflow of collecting performance data in an SGLang Ascend NPU inference service — from preparation, collection, and analysis to visualization — helping you get started with performance profiling quickly. For more profiling scenarios (e.g., Nsight Systems, PD disaggregation, etc.), see [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling). ## Ascend PyTorch Profiler SGLang has built-in PyTorch Profiler support. Through the Ascend `torch_npu` backend, you can directly collect NPU operator-level performance data. No additional packages are required — profiling start/stop is controlled via API requests. ### 1. Environment Setup Launch an SGLang online service and set the `SGLANG_TORCH_PROFILER_DIR` environment variable to control where performance files are saved. Once the service starts, profiling is ready on standby. ```shell Command theme={null} # Set the performance data output directory export SGLANG_TORCH_PROFILER_DIR=./sglang_profile # Start SGLang server (use local model path or HuggingFace model id) sglang serve \ --model-path /path/to/your/model \ --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 \ --max-running-requests 128 ``` **Profiling-related environment variables:**
Variable Description Default
SGLANG\_TORCH\_PROFILER\_DIR Trace file output directory /tmp
SGLANG\_PROFILE\_WITH\_STACK Record Python call stack (True / False) True
SGLANG\_PROFILE\_RECORD\_SHAPES Record operator input shapes (True / False) True
Priority order: **API parameter > Environment variable > Default value**. ### 2. Collection Methods SGLang provides four collection methods. The core differences are **whether you need to manually send `/start_profile` and `/stop_profile`**. All four methods produce identical results — choose the most convenient one. **Method comparison:**
Method Manual start\_profile Manual stop\_profile Notes
A: API manual start/stop Yes Yes Maximum flexibility for precise control
B: API auto-stop Yes No Set num\_steps, auto-stops and generates output
C: bench\_serving --profile No No Benchmark + profiling in one command
D: sglang.profiler CLI No No Standalone profiling CLI tool
#### Method A: API Manual Start/Stop Send `/start_profile` to start → send workload requests → send `/stop_profile` to stop. After stopping, the server automatically parses the data — **no need to manually call `analyse()`**. ```bash Command theme={null} # Step 1: Start profiling (no num_steps, requires manual stop) curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "./sglang_profile", "start_step": 1, "activities": ["CPU", "GPU"] }' # Step 2: Send workload requests (using curl as example) curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 10}}' # Step 3: Stop profiling curl -X POST http://127.0.0.1:30000/stop_profile ``` `/stop_profile` returns `"Stop profiling. This will take some time."` — the server needs time to flush trace data to disk and parse it. Wait for the response to complete. This method takes a significant amount of time to parse profiling data; consider using **Method B** instead to avoid lengthy waits. #### Method B: API Auto-Stop Specify `num_steps` in the `/start_profile` request. Profiling stops automatically after N steps and generates output — **no need to manually send `/stop_profile`**. ```bash Command theme={null} # num_steps=10, wait 3 warmup steps, auto-stop after 10 steps curl -X POST http://127.0.0.1:30000/start_profile \ -H "Content-Type: application/json" \ -d '{ "output_dir": "./sglang_profile", "start_step": 3, "num_steps": 10, "activities": ["CPU", "GPU"] }' # Just send workload — no /stop_profile needed curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}' ``` #### Method C: bench\_serving --profile Use SGLang's built-in `bench_serving` with the `--profile` flag. **Automatically handles `/start_profile` and `/stop_profile`** — no manual API calls needed. ```bash Command theme={null} # With --profile-steps: auto-stops after N steps and generates output python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 100 \ --num-prompts 10 \ --profile \ --profile-steps 10 \ --profile-output-dir ./sglang_profile # Without --profile-steps: /stop_profile sent automatically after benchmark python -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 100 \ --num-prompts 10 \ --profile \ --profile-output-dir ./sglang_profile ``` `--profile-steps N` sends `"num_steps": N` to the server's `/start_profile`, so the server auto-stops and parses data after N steps — bench\_serving skips sending `/stop_profile`. `bench_serving --profile` creates a timestamp subdirectory inside `--profile-output-dir` (e.g., `//`). The output path is shown in the server log as `Profiling done. Traces are saved to: `. On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically redirect PyTorch Profiler's CUDA activity to NPU, so `activities: ["CPU", "GPU"]` actually captures NPU operator events. **`bench_serving --profile` parameters:**
ParameterDescription
--profileEnable auto profiling start/stop
--profile-steps NAuto-stop after N steps (skips /stop\_profile)
--profile-output-dirTrace output directory
#### Method D: sglang.profiler CLI Use the `sglang.profiler` CLI module, which automatically sends `/start_profile` and waits for completion. **Start `sglang.profiler` first, then send inference requests** (otherwise there are no steps to capture and the profiler will wait indefinitely). ```bash Command theme={null} # Terminal 1: Start sglang.profiler first (sends /start_profile, then waits for completion) python3 -m sglang.profiler \ --url http://127.0.0.1:30000 \ --output-dir ./my_profiles \ --num-steps 3 \ --cpu --gpu & ``` ```bash Command theme={null} # Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests. # The profiler starts recording once /start_profile is received by the server. # Requests sent before the server receives /start_profile will not be captured. curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}' ``` A simpler and more reliable approach is to use `bench_serving --profile`, which handles both steps automatically: ```bash Command theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --base-url http://127.0.0.1:30000 \ --model /path/to/your/model \ --tokenizer /path/to/your/model \ --dataset-name random \ --random-input-len 128 \ --random-output-len 32 \ --num-prompts 10 \ --profile \ --profile-steps 3 \ --profile-output-dir ./my_profiles ``` `sglang.profiler` is essentially a CLI wrapper around the `/start_profile` API. Advanced options like `--profile-by-stage` are also supported. On Ascend NPU, trace flushing is asynchronous and may take a while — the CLI may occasionally block waiting for flush. If it times out, use Method B (API auto-stop) or Method C (bench\_serving --profile) instead. **`sglang.profiler` CLI parameters:**
ParameterDescription
--urlSGLang server address
--output-dir Output directory (defaults to SGLANG\_TORCH\_PROFILER\_DIR)
--num-stepsNumber of steps to profile
--profile-by-stage Profile prefill / decode stages separately
--profile-prefixTrace filename prefix
--cpu / --gpu / --mem / --rpd Activity types to collect
### 3. Full Parameter Reference All methods ultimately send a `/start_profile` request to the server. The full set of supported parameters:
Parameter Description Default
output\_dir Output directory. Falls back to SGLANG\_TORCH\_PROFILER\_DIR or /tmp /tmp
num\_steps Number of steps. If set, profiling auto-stops — no /stop\_profile needed None
start\_step Step index to start profiling (inclusive), for skipping warmup 0
activities Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, only CPU and GPU are supported. MEM depends on CUDA memory APIs and will be silently ignored. RPD requires ROCm and will cause an error. \["CPU", "GPU"]
profile\_by\_stage Profile prefill and decode stages separately false
with\_stack Record Python call stack. Also controllable via SGLANG\_PROFILE\_WITH\_STACK true
record\_shapes Record operator input shapes. Also controllable via SGLANG\_PROFILE\_RECORD\_SHAPES true
profile\_prefix Prefix for trace filenames None
profile\_stages Stages to profile, e.g., \["prefill", "decode"]. Requires profile\_by\_stage None
### 4. Finding Output Files **The server log explicitly indicates where traces are saved.** You can find them via: * **When profiling starts**: server log outputs `Profiling starts. Traces will be saved to: (with profile id: )` ```text theme={null} [2026-05-19 13:23:15] Profiling starts. Traces will be saved to: /tmp/1779196995.6948605 (with profile id: 1779196995.6979997) [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] INFO: 127.0.0.1:40714 - "POST /start_profile HTTP/1.1" 200 OK ``` * **When profiling stops**: server log outputs `Profiling done. Traces are saved to: ` ```text theme={null} [2026-05-19 13:23:17] Stop profiling... [2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data. [rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator()) [2026-05-19 13:23:17] [INFO] [352725] profiler.py: Start parsing profiling data: /tmp/1779196995.6948605/localhost.localdomain_350443_20260519132315700_ascend_pt [2026-05-19 13:23:22] [INFO] [352734] profiler.py: CANN profiling data parsed in a total time of 0:00:04.022310 [2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669 [2026-05-19 13:23:32] Profiling done. Traces are saved to: /tmp/1779196995.6948605 ``` * **CLI output**: `sglang.profiler` outputs `Dump profiling traces to ` ```text theme={null} Dump profiling traces to /tmp/1779243331.3219 Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False) ``` The directory structure is `/___ascend_pt/`. When using Method C (`bench_serving --profile`), a timestamp subdirectory is added: `//___ascend_pt/`. Always check the server log for the exact path: `Profiling done. Traces are saved to: `. ### 5. Viewing Results After profiling stops (either `/stop_profile` returns or `num_steps` auto-triggers), the server **automatically parses the raw data**. The `ASCEND_PROFILER_OUTPUT` directory directly contains the following visualization files — **no need to manually call `analyse()`**:
FileDescription
trace\_view\.json Chrome Tracing format. Open in MindStudio Insight
analysis.dbDatabase-format performance data
ascend\_pytorch\_profiler\_0.db Database-format performance data
kernel\_details.csvKernel-level data
operator\_details.csvOperator-level data
step\_trace\_time.csvStep trace timing data
`trace_view.json` can also be opened using Chrome's built-in `chrome://tracing` or [Perfetto UI](https://ui.perfetto.dev/). In a multi-node deployment, each node generates its own trace files. To merge them into a single unified trace, set `"merge_profiles": true` in the `/start_profile` request. However, on Ascend NPU, the merge feature does **not** fully support the `*_ascend_pt` format — merged results may be incomplete or incorrect. It is recommended to view `trace_view.json` on each node individually instead. For more details, see [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces). ### 6. Re-parsing Raw Data (Optional) If you need to **re-parse existing data** with different parameters, or if profiling was interrupted and `ASCEND_PROFILER_OUTPUT` was not auto-generated, use `torch_npu`'s `analyse()` tool: ```python theme={null} from torch_npu.profiler.profiler import analyse analyse("./sglang_profile/_*_ascend_pt/") ``` Normally **no need** to manually run `analyse()` — the server already parses data automatically. Only use this for re-parsing or handling interrupted data. Running `analyse()` when ASCEND\_PROFILER\_OUTPUT already exists will overwrite the original directory. If the original data is still needed, back it up before running `analyse()`. ## Best Practices ### Common Notes * **Finding output**: Check the server log for `Profiling starts. Traces will be saved to: ` and `Profiling done. Traces are saved to: `, or `sglang.profiler` output for `Dump profiling traces to `. * **Control trace file size**: Reduce the number of requests and output length using `--num-prompts` and `--random-output-len` to avoid trace files too large for browsers. * **Warmup iterations**: Set `start_step` to skip the first few warmup steps and capture performance data under steady state. * **Profile step count**: Large values for `num_steps` or `--profile-steps` can lead to lengthy profiling data parsing times. Reduce these values appropriately when you only need a quick overview. * **CUDA Graph impact**: To see the full Python call stack → operator mapping in traces, add `--disable-cuda-graph` when starting the server. Note that this reduces decode performance — only use during profiling. To analyze CUDA Graph capture specifically, use `--enable-profile-cuda-graph` — traces are saved to `SGLANG_TORCH_PROFILER_DIR/graph_capture_profile/`. * **Multi-node deployment**: In multi-node environments, performance data is distributed across nodes. On Ascend NPU, the `merge_profiles` feature has limited support — check `*_ascend_pt/ASCEND_PROFILER_OUTPUT/trace_view.json` on each node individually. In PD disaggregation mode, prefill and decode workers must be profiled separately — see [Profile In PD Disaggregation Mode](/docs/developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode). ## See Also * [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling) — General SGLang profiling guide * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — Ascend NPU environment setup * [Ascend NPU Optimization](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning) — Ascend NPU optimization parameters * [Ascend NPU Performance Testing](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing) — Ascend NPU performance benchmarking * [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables) — Environment variable reference # Quantization on Ascend Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/quantization To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. [ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
Quantization scheme Layer type A2 Supported A3 Supported Ascend 950 Products Supported Diffusion models
W4A4 dynamic Linear √ √ TBD √
W8A8 static Linear √ √ TBD √
W8A8 dynamic Linear √ √ TBD √
MXFP8 (Diffusion, LLM dense) Linear x x √ √
MXFP4 Linear x x √ √
MXFP4 W4A8 Linear x x √ x
MXFP4 W4A8 (ModelSlim) MoE x x √ x
MXFP4 W4A4 Linear x x WIP x
MXFP4 W4A4 (ModelSlim) MoE x x √ x
W4A4 dynamic MoE √ √ TBD x
W4A8 dynamic MoE √ √ TBD x
W8A8 dynamic MoE √ √ TBD x
MXFP8 (LLM MoE) MoE x x √ x
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
Quantization scheme Layer type A2 Supported A3 Supported Ascend 950 Products Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MoE √ √ TBD
GPTQ on Ascend support
Quantization scheme Layer type A2 Supported A3 Supported Ascend 950 Products Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MOE MoE √ √ TBD
W8A16 MOE MoE √ √ TBD
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
Quantization scheme Layer type A2 Supported A3 Supported Ascend 950 Products Supported
W4A16 Linear √ √ TBD
W8A16 Linear √ √ TBD
W4A16 MoE √ √ TBD
W8A16 MoE √ √ TBD
Compressed-tensors (LLM Compressor) on Ascend support:
Quantization scheme Layer type A2 Supported A3 Supported Ascend 950 Products Supported
W8A8 dynamic Linear √ √ TBD
W4A8 dynamic with/without activation clip MoE √ √ TBD
W4A16 MOE MoE √ √ TBD
W8A8 dynamic MoE √ √ TBD
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
Quantization type Layer type A2 Supported A3 Supported Ascend 950 Products Supported
All GGUF types (standard, K-quant) Linear √ √ TBD
All GGUF types (standard, K-quant) MoE √ √ TBD
**Usage Examples:** * Dense model (e.g., Qwen3-14B-Q4\_K\_M.gguf): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen3-14B-Q4_K_M.gguf \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.7 --tp-size 2 ``` * MoE model (e.g., Qwen3-30B-A3B-Q4\_K\_M.gguf): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen3-30B-A3B-Q4_K_M.gguf \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 2 ``` > **Implementation Notes:** > > * GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead). > * MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation. > * TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models. **MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950 Products or newer (`npu_dynamic_mx_quant` is not available on A2 / A3). * Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` > **Implementation Notes:** > > * Online path: `Fp8Config.get_quant_method()` dispatches to `NPUMXFP8LinearMethod`. Weights are quantized once at load via `npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn)` and pre-transposed to `[in, out]`; activations are per-token quantized at inference and matmul runs via `npu_quant_matmul(..., group_sizes=[1, 1, 32])` (block\_size = 32). > * Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping. > * MoE MXFP8 (FusedMoE) for LLMs is documented in **MXFP8 for LLM MoE models** below. **MXFP8 for LLM MoE models (e.g. Qwen3-30B-A3B / Qwen3.5 MoE):** LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend A5 series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are A5-only. * Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-30B-A3B \ --quantization mxfp8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme). No `--quantization` flag is needed: the `quant_model_description.json` shipped with the checkpoint selects both the ModelSlim path and the scheme automatically. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path /path/to/Qwen3-30B-A3B-W8A8-MXFP8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` > **Implementation Notes:** > > * Both paths share the per-gmm kernel `NPUMXFP8MoEMethod` (`hardware_backend/npu/quantization/moe_methods.py`), which tells online from offline by weight dtype. Expert weights and their e8m0 block scales are kept as non-contiguous transpose views — calling `.contiguous()` would tank HBM bandwidth. > * Online path: `Fp8Config.get_quant_method()` dispatches FusedMoE layers to `NPUMXFP8OnlineMoEMethod`, which subclasses `UnquantizedFusedMoEMethod` and overrides only `create_moe_runner` to swap in the MXFP8 kernels — weight creation, weight post-processing and the forward pass are the unquantized Ascend ones. BF16 expert weights `w13`/`w2` are quantized once at load via `npu_dynamic_mx_quant(dst_type=torch.float8_e4m3fn)` (a 3D `[E, N, K]` input is accepted directly). > * Offline path: `ModelSlimMXFP8MoEScheme` (one instance per weight group) loads `float8_e4m3fn` expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped `[E, N, K/32] → [E, N, K/64, 2]` (contiguous pairing, matching `npu_dynamic_mx_quant`) then transposed. > * Forward: `AscendTPDispatcher` runs `npu_moe_init_routing_v2(quant_mode=3)`, which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). `AscendRunnerCore` then runs gmm1 `npu_grouped_matmul_swiglu_quant_v2` (cumulative `group_list`; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 `npu_grouped_matmul` (count `group_list`). The UE8M0 (`float8_e8m0fnu`) scale dtypes are passed explicitly; the e4m3 `x`/`weight` dtypes are left implicit. > * **Router gate**: msmodelslim may also quantize `mlp.gate` (`W8A8_MXFP8`). The gate is a `ReplicatedLinear`, so its quantization must be **description-driven**: for the offline `modelslim` path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output. > * Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on Ascend A5. **MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950 Products or newer. * Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp_w4a8 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline W4A8 quantization (msmodelslim pre-quantized weights, `W4A8_MXFP` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`). > **Implementation Notes:** > > * Weights are packed FP4 (`float4_e2m1fn_x2`, two nibbles per byte) with a UE8M0 per-block shared exponent (block\_size = 32); activations are per-token MXFP8. Matmul runs via `npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32])`. > * The packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype` / `input_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject. > * Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration). **ModelSlim W4A8 MXFP4 for LLM MoE models:** SGLang auto-detects offline ModelSlim `W4A8_MXFP` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend A5 or newer. ```bash Command theme={null} MODEL_PATH=/path/to/w4a8-mxfp4-moe-model python3 -m sglang.launch_server \ --model-path "$MODEL_PATH" \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 ``` > **Implementation Notes:** > > * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32). > * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP8 dispatch. SGLang dynamically quantizes each expert input to MXFP8 immediately before grouped matmul. **MXFP4 W4A4 for LLM dense models (e.g. Qwen3 / Qwen3.5):** LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend A5 series (Ascend 950) or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead). * Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time): ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --quantization mxfp4 \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --mem-fraction-static 0.8 --tp-size 1 ``` * Offline W4A4 quantization (msmodelslim pre-quantized weights, `W4A4_MXFP4` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`). > **Implementation Notes:** > > * **Online** (`NPUDualLevelMXFP4LinearMethod`) uses **dual-level** MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via `npu_dynamic_dual_level_mx_quant`, and the matmul runs via `npu_dual_level_quant_matmul` (weight in FRACTAL\_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS). > * **Offline** (`ModelSlimMXFP4Scheme` → `NPUSingleLevelMXFP4OfflineLinearMethod`) is **single-level**: msmodelslim's `W4A4_MXFP4` checkpoint ships single-level UE8M0 block scales (block\_size = 32), so the matmul runs via `npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32])`. The online and offline paths therefore use different matmul kernels — they no longer share the matmul path. > * As with W4A8, the packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject. > * Validated end-to-end on Ascend A5 hardware. **ModelSlim W4A4 MXFP4 for LLM MoE models:** SGLang auto-detects offline ModelSlim `W4A4_MXFP4` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950 products or newer. ```bash Command theme={null} MODEL_PATH=/path/to/w4a4-mxfp4-moe-model python3 -m sglang.launch_server \ --model-path "$MODEL_PATH" \ --device npu --attention-backend ascend \ --host 0.0.0.0 --port 30000 \ --tp-size 1 ``` > **Implementation Notes:** > > * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32). > * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP4 dispatch. SGLang dynamically quantizes each expert input to MXFP4 immediately before grouped matmul. ## Diffusion Model Quantization on Ascend NPU SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950 Products; the ModelSlim W8A8/W4A4 schemes work on A2/A3. **Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950 Products
Quantization method quant\_type in JSON Scheme class Mode A2/A3 Supported Ascend 950 Products Supported Trigger
MXFP8 (W8A8) — MXFP8Config Online x √ --quantization mxfp8
MXFP8 (W8A8) W8A8\_MXFP8 ModelSlimMXFP8Scheme Offline x √ auto-detected from quant\_model\_description.json
W8A8 static W8A8 ModelSlimW8A8Int8 Offline √ TBD auto-detected from quant\_model\_description.json
W8A8 dynamic W8A8\_DYNAMIC ModelSlimW8A8Int8 Offline √ TBD auto-detected from quant\_model\_description.json
W4A4 dynamic W4A4\_DYNAMIC ModelSlimW4A4Int4 Offline √ TBD auto-detected from quant\_model\_description.json
### Online MXFP8 Quantization Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection. ```bash Command theme={null} # Start the diffusion server with online MXFP8 quantization sglang serve \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --num-gpus 4 ``` ```bash Command theme={null} # One-shot generation sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --prompt "a beautiful sunset over the mountains" \ --save-output ``` ### Offline MXFP8 Quantization (ModelSlim) For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed. **Step 1: Quantize with msModelSlim** ```bash Command theme={null} msmodelslim quant \ --model_path /path/to/wan2_2_float_weights \ --save_path /path/to/wan2_2_mxfp8_weights \ --device npu \ --model_type Wan2_2 \ --quant_type mxfp8 \ --trust_remote_code True ``` > Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim. **Step 2: Convert to Diffusers format** msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script: ```bash Command theme={null} python python/sglang/multimodal_gen/tools/wan_repack.py \ --input-path /path/to/wan2_2_mxfp8_weights \ --output-path /path/to/wan2_2_mxfp8_diffusers ``` Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory. **Step 3: Run inference** ```bash Command theme={null} sglang generate \ --model-path /path/to/wan2_2_mxfp8_diffusers \ --prompt "a beautiful sunset over the mountains" \ --save-output ``` For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). # Ring-SP Performance (Wan2.1-T2V-1.3B) Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`. * Baseline config: `ulysses=1, ring=1` (short: `u1r1`) * Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`) ## Benchmark Setup * Model: `Wan2.1-T2V-1.3B-Diffusers` * Prompt: `"a cat is playing piano"` * Framework command: `sglang generate` * Runtime: `torch_npu==2.10.0` ## Generate Commands ### Baseline (`u1r1`) ```bash theme={null} sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \ --prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \ --save-output ``` ### Ring-SP (`u1r2`) ```bash theme={null} sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \ --prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \ --save-output ``` ## Benchmarks Benchmark Disclaimer These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload. ### Stage Time Breakdown
Stage / Metric u1r2 (s) u1r1 baseline (s) Speedup
InputValidation 0.0003 0.0002 0.67x
TextEncoding 3.5936 3.5820 1.00x
LatentPreparation 0.0007 0.0055 7.86x
TimestepPreparation 0.0008 0.0007 0.88x
Denoising 121.2788 239.2580 1.97x
Decoding 13.8685 16.4969 1.19x
Total (Pixel data generated) 141.86 266.50 1.88x
## Summary * With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case. * End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`). * The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`). # Environment Variables Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/environment_variables SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior. This document provides a list of commonly used environment variables and aims to stay updated over time. ## Directly Used in SGLang
Environment Variable Description Default Value
SGLANG\_NPU\_USE\_MLAPO Adopts the MLAPO fusion operator in the attention
preprocessing stage of the MLA model.		 false
SGLANG\_USE\_FIA\_NZ Reshapes KV Cache for FIA NZ format.
SGLANG\_USE\_FIA\_NZ must be enabled with SGLANG\_NPU\_USE\_MLAPO		 false
SGLANG\_NPU\_USE\_MULTI\_STREAM Enable dual-stream computation of shared experts
and routing experts in DeepSeek models.
Enable dual-stream computation in DeepSeek DSA Indexer.		 false
SGLANG\_NPU\_DISABLE\_ACL\_FORMAT\_WEIGHT Disable casting model weight tensor to a specific NPU
ACL format.		 false
SGLANG\_DEEPEP\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK The maximum number of dispatched tokens on each rank. 128
## Used in DeepEP Ascend
Environment Variable Description Default Value
DEEPEP\_NORMAL\_LONG\_SEQ\_PER\_ROUND\_TOKENS Enable long-sequence token pipelining in dispatch stage. Indicates
the number of tokens transmitted per round on each rank.		 8192
DEEPEP\_NORMAL\_LONG\_SEQ\_ROUND Enable long-sequence token pipelining in dispatch stage. Indicates
the number of rounds transmitted on each rank.		 1
DEEPEP\_NORMAL\_COMBINE\_ENABLE\_LONG\_SEQ Enable long-sequence token pipelining in combine stage.
The value 0 means disabled.		 0
MOE\_ENABLE\_TOPK\_NEG\_ONE Needs to be enabled when the expert ID to be processed by
DEEPEP contains -1.		 0
DEEP\_NORMAL\_MODE\_USE\_INT8\_QUANT Deprecated — will be removed in a future release.
When set to 1, quantizes intermediate activations to INT8 in
the DeepEP dispatch operator during normal mode, reducing communication
volume for W8A8-quantized MoE models. This variable will become a no-op;
the quantization behavior will be inferred automatically. 		0
DEEPEP\_HCCL\_BUFFSIZE Configures the HCCL buffer size (in MB) for the process groups used by
DeepEP MoE All-to-All communication (the default group and moe-named
groups), allowing the MoE communication buffer to be tuned independently from
non-MoE groups. If unset, falls back to HCCL\_BUFFSIZE, then to 200.
A larger buffer reduces All-to-All latency at the cost of more HBM usage.		 200
## Others
Environment Variable Description Default Value
TASK\_QUEUE\_ENABLE Used to control the optimization level of the dispatch queue
for the task\_queue operator. Detail		 1
INF\_NAN\_MODE\_ENABLE Controls whether the chip uses saturation mode or INF\_NAN mode. Detail 1
STREAMS\_PER\_DEVICE Configures the maximum number of streams for the stream pool. Detail 32
PYTORCH\_NPU\_ALLOC\_CONF Controls the behavior of the cache allocator.
This variable changes memory usage and may cause performance fluctuations. Detail
ASCEND\_MF\_STORE\_URL The address of config store in MemFabric during PD separation,
which is generally set to the IP address of the P primary node
with an arbitrary port number.
ASCEND\_LAUNCH\_BLOCKING Controls whether synchronous mode is enabled during operator execution. Detail 0
HCCL\_OP\_EXPANSION\_MODE Configures the expansion position for communication algorithm scheduling. Detail
HCCL\_BUFFSIZE Controls the size of the buffer area for shared data between two NPUs.
The unit is MB, and the value must be greater than or equal to 1. Detail		 200
HCCL\_SOCKET\_IFNAME Configures the name of the network card used by the Host
during HCCL initialization. Detail
GLOO\_SOCKET\_IFNAME Configures the network interface name for GLOO communication.
# Glossary Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/glossary This page covers the hardware concepts, communication libraries, deployment terminology, and common abbreviations you will encounter throughout the Ascend NPU documentation. Refer back here when you run into unfamiliar terms. ## Hardware ### Supported devices SGLang supports the following Ascend inference hardware: | Hardware | Chip | Devices | Dies per card | Memory configuration | | ------------------ | ----------- | ------- | ------------- | ------------------------------------ | | Atlas 800I A2 (A2) | Ascend 910B | 8 | 1 | 8(cards) × 1(die/card) × 64(GB/die) | | Atlas 800I A3 (A3) | Ascend 910C | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) | Throughout these docs, **A2** and **A3** are used as shorthand for the hardware above. Docker image tags use `910b` for A2 and `a3` for A3. For example, `cann9.0.0-910b-v0.5.16` and `cann9.0.0-a3-v0.5.16`. On A3, each card has 2 dies, giving 16 devices vs 8 on A2. Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on A3 means `--tp-size 8`. From a deployment perspective, the two key differences between A2 and A3 are: 1. dies per card — which drives both total memory and `--tp-size` configuration 2. PD disaggregation — A2 requires setting `export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"`, while A3 uses the default protocol. ### NPU **NPU** stands for Neural Processing Unit. Each NPU device is a single `davinci` core. The terms "NPU" and "davinci" are used interchangeably in commands and error logs. On A2, devices are numbered `/dev/davinci0` through `/dev/davinci7` (8 devices). On A3, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices). On either an A2 or A3 server, run `npu-smi info` to view NPU information such as device health, memory usage, and chip status. If the command is not found or reports no devices, the driver is likely not installed. Follow the [Ascend driver installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0005.html?OS=openEuler\&InstallType=local) to install it. ## Communication libraries | Library | Description | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **HCCL** (Huawei Collective Communication Library) | The primary communication backend for multi-card data transfer on Ascend NPUs. Equivalent to NVIDIA NCCL. Used in `--nnodes`, `--tp-size`, and all distributed scenarios. | | **GLOO** | Meta's collective communications library. Used alongside HCCL for distributed initialization and coordination. | | **DeepEP** (Deep Expert Parallelism) | A communication library optimized for Mixture-of-Experts (MoE) all-to-all dispatch and combine operations. Used with `--moe-a2a-backend deepep`. | | **RDMA** (Remote Direct Memory Access) | Enables direct memory access between nodes over InfiniBand or RoCE networks. Required for multi-node PD disaggregation. | ## Quantization and precision | Notation | Meaning | | --------- | ------------------------------------------------------------------ | | **W8A8** | 8-bit weights, 8-bit activations | | **W4A8** | 4-bit weights, 8-bit activations | | **W4A16** | 4-bit weights, 16-bit activations | | **BF16** | Brain Floating Point 16 — 16-bit format optimized for ML workloads | | **FP8** | 8-bit Floating Point — not supported on A2/A3 | | **INT8** | 8-bit Integer quantization | To apply quantization, use `--quantization modelslim` for W8A8 INT8, or load a pre-quantized checkpoint directly from a model hub. ## Deployment terminology ### Prefill-Decode (PD) disaggregation PD disaggregation separates inference into two stages running on different nodes: * **Prefill** (P): Processes the entire input prompt at once. Compute-bound. * **Decode** (D): Generates tokens one at a time. Memory-bandwidth-bound. **PD Mixed**: Both stages run on the same set of nodes. | Shorthand | Meaning | | --------- | ------------------------------- | | **1P1D** | 1 prefill node + 1 decode node | | **2P1D** | 2 prefill nodes + 1 decode node | | **1P2D** | 1 prefill node + 2 decode nodes | You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) section headings, e.g., `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes + 1 decode node, 32 cards total." ### Parallelism strategies | Strategy | Flag | Description | | ----------------------------- | ---------------- | ----------------------------------------------------------------------- | | **Tensor Parallelism (TP)** | `--tp-size` | Splits model weights across NPUs within a node | | **Data Parallelism (DP)** | `--dp-size` | Replicates the model across nodes for higher throughput | | **Expert Parallelism (EP)** | `--ep-size` | Distributes MoE experts across devices; requires `--moe-a2a-backend` | | **Context Parallelism (CP)** | `--attn-cp-size` | Splits long context windows across devices for extended sequence length | | **Pipeline Parallelism (PP)** | `--pp-size` | Splits model layers across devices sequentially | ### Speculative decoding | Algorithm | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **EAGLE3** | Uses an external draft model (specified via `--speculative-draft-model-path`) to propose candidate tokens. Supports top-k sampling. | | **NEXTN** | Uses the model's built-in Multi-Token Prediction (MTP) heads — no separate draft model needed. Available for models with native MTP support. | | **MTP** (Multi-Token Prediction) | A model architecture feature where the model predicts multiple future tokens per step. The foundation for NEXTN speculative decoding. | ### Performance metrics | Metric | Description | | -------------------------------- | ------------------------------------------------------------------------------------ | | **TPOT** (Time Per Output Token) | Average time to generate each output token. Lower is better. | | **TTFT** (Time To First Token) | Latency from request arrival to first generated token. Critical for interactive use. | ## Model architecture terms | Term | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **MoE** (Mixture of Experts) | Model architecture where only a subset of parameters (experts) is activated per token, reducing compute. Common in DeepSeek, Qwen3-30B-A3B, and MiMo models. | | **MLA** (Multi-head Latent Attention) | Attention variant that compresses key-value representations into a lower-dimensional latent space. Used by DeepSeek models. | | **GQA** (Grouped-Query Attention) | Attention variant where multiple query heads share a single key-value head. Used by Qwen3 dense models. | | **DSA** (DeepSeek Sparse Attention) | DeepSeek's sparse attention mechanism; reduces KV cache overhead for long contexts. | | **FFN** (Feed-Forward Network) | The non-attention component of each transformer layer. In MoE models, the FFN is replaced by multiple expert FFN layers selected by a router. | ## Other common abbreviations | Abbreviation | Expansion | | ------------------------------------------------ | ----------------------------------------------------------------------- | | **ACL** (Ascend Computing Language) | Low-level Ascend compute API; seen in error logs | | **DVFS** (Dynamic Voltage and Frequency Scaling) | Hardware frequency scaling to maintain performance stability | | **NUMA** (Non-Uniform Memory Access) | Memory architecture affecting multi-socket performance | | **KV Cache** (Key-Value Cache) | Cached attention key-value tensors to avoid recomputation during decode | | **LoRA** (Low-Rank Adaptation) | Parameter-efficient fine-tuning method | | **HF** (Hugging Face) | Model hub; `HF_TOKEN` / `HF_ENDPOINT` control model download access | | **UB** (Unified Buffer) | On-chip NPU memory; referenced in operator optimization | | **DMA** (Direct Memory Access) | Data transfer mechanism between host and device memory | ## Where to go next * [Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — launch your first server * [Installation Guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) — full installation with component version mapping * [Supported Features](/docs/hardware-platforms/ascend-npus/reference/support_features) — per-parameter Ascend support status * [Supported Models](/docs/hardware-platforms/ascend-npus/reference/support_models) — models verified on Ascend NPU * [Model Tutorials](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1) — step-by-step deployment guides * [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — benchmark configurations and results # Supported Features Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_features This section describes the basic functions and features supported by the Ascend NPU. If you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues). If you want to know the meaning and usage of each parameter, click [Server Arguments](../../../advanced_features/server_arguments). ## Model and tokenizer
Argument Defaults Options Server supported
`--model-path`
`--model`		 `None` Type: str A2, A3
`--tokenizer-path` `None` Type: str A2, A3
`--tokenizer-mode` `auto` `auto`, `slow` A2, A3
`--tokenizer-backend` `huggingface` `huggingface`, `fastokens` A2, A3
`--tokenizer-worker-num` `1` Type: int A2, A3
`--detokenizer-worker-num` `1` Type: int A2, A3
`--skip-tokenizer-init` `False` bool flag (set to enable) A2, A3
`--load-format` `auto` `auto`, `safetensors`, `gguf` A2, A3
`--model-loader-extra-config` `{}` Type: str A2, A3
`--trust-remote-code` `False` bool flag (set to enable) A2, A3
`--context-length` `None` Type: int A2, A3
`--is-embedding` `False` bool flag (set to enable) A2, A3
`--enable-multimodal` `None` bool flag (set to enable) A2, A3
`--revision` `None` Type: str A2, A3
`--model-impl` `auto` `auto`, `sglang`,
`transformers`		 A2, A3
`--model-config-parser` `auto` Type: str A2, A3
## HTTP server
Argument Defaults Options Server supported
`--host` `127.0.0.1` Type: str A2, A3
`--port` `30000` Type: int A2, A3
`--skip-server-warmup` `False` bool flag (set to enable) A2, A3
`--warmups` `None` Type: str A2, A3
`--nccl-port` `None` Type: int A2, A3
`--fastapi-root-path` `None` Type: str A2, A3
`--grpc-mode` `False` bool flag (set to enable) Planned
## SSL/TLS
Argument Defaults Options Server supported
`--ssl-keyfile` `None` Type: str A2, A3
`--ssl-certfile` `None` Type: str A2, A3
`--ssl-keyfile-password` `None` Type: str A2, A3
`--enable-ssl-refresh` `False` bool flag
(set to enable)		 A2, A3
`--enable-http2` `False` bool flag
(set to enable)		 A2, A3
## Quantization and data type
Argument Defaults Options Server supported
`--dtype` `auto` `auto`,
`float16`,
`bfloat16`		 A2, A3
`--quantization` `None` `modelslim` A2, A3
`--quantization-param-path` `None` Type: str Special for GPU
`--kv-cache-dtype` `auto` `auto` A2, A3
`--enable-fp32-lm-head` `False` bool flag
(set to enable)		 A2, A3
`--modelopt-quant` `None` Type: str Special for GPU
`--modelopt-checkpoint-restore-path` `None` Type: str Special for GPU
`--modelopt-checkpoint-save-path` `None` Type: str Special for GPU
`--modelopt-export-path` `None` Type: str Special for GPU
`--quantize-and-serve` `False` bool flag
(set to enable)		 Special for GPU
`--rl-quant-profile` `None` Type: str Special for GPU
## Memory and scheduling
Argument Defaults Options Server supported
`--mem-fraction-static` `None` Type: float A2, A3
`--max-running-requests` `None` Type: int A2, A3
`--prefill-max-requests` `None` Type: int A2, A3
`--max-queued-requests` `None` Type: int A2, A3
`--max-total-tokens` `None` Type: int A2, A3
`--chunked-prefill-size` `None` Type: int A2, A3
`--max-prefill-tokens` `16384` Type: int A2, A3
`--schedule-policy` `fcfs` `lpm`,
`fcfs`,
`random`		 A2, A3
`--enable-priority-scheduling` `False` bool flag
(set to enable)		 A2, A3
`--disable-priority-preemption` `False` bool flag
(set to enable)		 A2, A3
`--default-priority-value` `None` Type: int A2, A3
`--schedule-low-priority-values-first` `False` bool flag
(set to enable)		 A2, A3
`--priority-scheduling-preemption-threshold` `10` Type: int A2, A3
`--retraction-policy` `length` `length`,
`priority`		 A2, A3
`--schedule-conservativeness` `1.0` Type: float A2, A3
`--page-size` `128` Type: int A2, A3
`--swa-full-tokens-ratio` `0.8` Type: float A2, A3
`--disable-hybrid-swa-memory` `False` bool flag
(set to enable)		 A2, A3
`--radix-eviction-policy` `lru` `lru`,
`lfu`		 A2, A3
`--enable-prefill-delayer` `False` bool flag
(set to enable)		 A2, A3
`--prefill-delayer-max-delay-passes` `30` Type: int A2, A3
`--prefill-delayer-token-usage-low-watermark` `None` Type: float A2, A3
`--prefill-delayer-forward-passes-buckets` `None` List\[float] A2, A3
`--prefill-delayer-wait-seconds-buckets` `None` List\[float] A2, A3
`--prefill-delayer-queue-min-ratio` `None` Type: float A2, A3
`--prefill-delayer-max-delay-ms` `None` Type: float A2, A3
`--abort-on-priority-when-disabled` `False` bool flag
(set to enable)		 A2, A3
`--enable-dynamic-chunking` `False` bool flag
(set to enable)		 A2, A3
## Runtime options
Argument Defaults Options Server supported
`--device` `None` Type: str A2, A3
`--tensor-parallel-size`
`--tp-size`		 `1` Type: int A2, A3
`--pipeline-parallel-size`
`--pp-size`		 `1` Type: int; Currently `2` not supported; Cannot be used together with TP A2, A3
`--attention-context-parallel-size`
`--attn-cp-size`		 `1` Type: int; must be equal to --tp-size A2, A3
`--moe-data-parallel-size`
`--moe-dp-size`		 `1` Type: int A2, A3
`--pp-max-micro-batch-size` `None` Type: int A2, A3
`--pp-async-batch-depth` `0` Type: int A2, A3
`--stream-interval` `1` Type: int A2, A3
`--batch-notify-size` `16` Type: int A2, A3
`--incremental-streaming-output` `False` bool flag (set to enable) A2, A3
`--stream-response-default-include-usage` `False` bool flag (set to enable) A2, A3
`--enable-streaming-session` `False` bool flag (set to enable) A2, A3
`--enable-session-radix-cache` `False` bool flag
(set to enable)		 A2, A3
`--random-seed` `None` Type: int A2, A3
`--constrained-json-whitespace-pattern` `None` Type: str A2, A3
`--constrained-json-disable-any-whitespace` `False` bool flag (set to enable) A2, A3
`--watchdog-timeout` `300` Type: float A2, A3
`--soft-watchdog-timeout` `300` Type: float A2, A3
`--dist-timeout` `None` Type: int A2, A3
`--download-dir` `None` Type: str A2, A3
`--model-checksum` `None` Type: str A2, A3
`--base-gpu-id` `0` Type: int A2, A3
`--gpu-id-step` `1` Type: int A2, A3
`--sleep-on-idle` `False` bool flag (set to enable) A2, A3
`--load-snapshot-publish-interval` `15` Type: int A2, A3
`--use-ray` `False` bool flag (set to enable) Special for GPU
`--custom-sigquit-handler` `None` Only for engine A2, A3
## Logging
Argument Defaults Options Server supported
`--log-level` `info` Type: str A2, A3
`--log-level-http` `None` Type: str A2, A3
`--log-requests` `False` bool flag
(set to enable)		 A2, A3
`--log-requests-level` `2` `0`, `1`, `2`, `3` A2, A3
`--log-requests-format` text `text`, `json` A2, A3
`--crash-dump-folder` `None` Type: str A2, A3
`--enable-metrics` `False` bool flag
(set to enable)		 A2, A3
`--enable-mfu-metrics` `False` bool flag
(set to enable)		 A2, A3
`--enable-metrics-for-all-schedulers` `False` bool flag
(set to enable)		 A2, A3
`--tokenizer-metrics-custom-labels-header` `x-custom-labels` Type: str A2, A3
`--tokenizer-metrics-allowed-custom-labels` `None` List\[str] A2, A3
`--extra-metric-labels` `None` Type: JSON/Dict A2, A3
`--bucket-time-to-first-token` `None` List\[float] A2, A3
`--bucket-inter-token-latency` `None` List\[float] A2, A3
`--bucket-e2e-request-latency` `None` List\[float] A2, A3
`--collect-tokens-histogram` `False` bool flag
(set to enable)		 A2, A3
`--prompt-tokens-buckets` `None` List\[str] A2, A3
`--generation-tokens-buckets` `None` List\[str] A2, A3
`--gc-warning-threshold-secs` `0.0` Type: float A2, A3
`--decode-log-interval` `40` Type: int A2, A3
`--enable-request-time-stats-logging` `False` bool flag
(set to enable)		 A2, A3
`--kv-events-config` `None` Type: str Special for GPU
`--enable-forward-pass-metrics` `False` bool flag (set to enable) A2, A3
`--forward-pass-metrics-worker-id` \`\` Type: str A2, A3
`--forward-pass-metrics-ipc-name` `None` Type: str A2, A3
`--enable-trace` `False` bool flag
(set to enable)		 A2, A3
`--trace-modules` `request` Type: str A2, A3
`--otlp-traces-endpoint` `localhost:4317` Type: str A2, A3
`--log-requests-target` `None` Type: str A2, A3
`--uvicorn-access-log-exclude-prefixes` `[]` List\[str] A2, A3
## RequestMetricsExporter configuration
Argument Defaults Options Server supported
`--export-metrics-to-file` `False` bool flag
(set to enable)		 A2, A3
`--export-metrics-to-file-dir` `None` Type: str A2, A3
## API related
Argument Defaults Options Server supported
`--api-key` `None` Type: str A2, A3
`--admin-api-key` `None` Type: str A2, A3
`--served-model-name` `None` Type: str A2, A3
`--weight-version` `default` Type: str A2, A3
`--chat-template` `None` Type: str A2, A3
`--hf-chat-template-name` `None` Type: str A2, A3
`--completion-template` `None` Type: str A2, A3
`--file-storage-path` `sglang_storage` Type: str Unused reserved parameter
`--enable-cache-report` `False` bool flag
(set to enable)		 A2, A3
`--reasoning-parser` `None` `deepseek-r1`
`deepseek-v3`
`glm45`
`gpt-oss`
`kimi`
`qwen3`
`qwen3-thinking`
`step3`		 A2, A3
`--default-chat-template-kwargs` `None` Type: JSON / Dict A2, A3
`--strip-thinking-cache` `False` bool flag (set to enable) A2, A3
`--enable-strict-thinking` `False` bool flag (set to enable) A2, A3
`--tool-call-parser` `None` `llama3`
`pythonic`
`qwen`
`qwen3_coder`		 A2, A3
`--sampling-defaults` `model` `openai`, `model` A2, A3
`--asr-max-buffer-seconds` `60` Type: int A2, A3
`--asr-max-concurrent-sessions` `32` Type: int A2, A3
## Data parallelism
Argument Defaults Options Server supported
`--data-parallel-size`
`--dp-size`		 `1` Type: int A2, A3
`--load-balance-method` `auto` `auto`,
`round_robin`,
`follow_bootstrap_room`,
`total_requests`,
`total_tokens`		 A2, A3
## Multi-node distributed serving
Argument Defaults Options Server supported
`--dist-init-addr`
`--nccl-init-addr`		 `None` Type: str A2, A3
`--nnodes` `1` Type: int A2, A3
`--node-rank` `0` Type: int A2, A3
## Model override args
Argument Defaults Options Server supported
`--json-model-override-args` `{}` Type: str A2, A3
`--preferred-sampling-params` `None` Type: str A2, A3
## LoRA (restricted to Qwen series models)
Argument Defaults Options Server supported
`--enable-lora` `False` bool flag
(set to enable)		 A2, A3
`--enable-lora-overlap-loading` `False` bool flag
(set to enable)		 A2, A3
`--max-lora-rank` `None` Type: int A2, A3
`--lora-target-modules` `None` `all` A2, A3
`--lora-paths` `None` Type: List\[str] /
JSON objects		 A2, A3
`--max-loras-per-batch` `8` Type: int A2, A3
`--max-loaded-loras` `None` Type: int A2, A3
`--lora-eviction-policy` `lru` `lru`,
`fifo`		 A2, A3
`--lora-backend` `csgmv` `triton`,
`csgmv`,
`ascend`,
`torch_native`		 A2, A3
`--experts-shared-outer-loras` `None` Type: bool A2, A3
`--lora-use-virtual-experts` `False` bool flag
(set to enable)		 Special for GPU
`--lora-strict-loading` `False` Type: bool A2, A3
`--lora-drain-wait-threshold` `0.0` Type: float A2, A3
`--max-lora-chunk-size` `16` `16`, `32`,
`64`, `128`		 Special for GPU
## Kernel backends (attention, sampling, grammar, GEMM)
Argument Defaults Options Server supported
`--attention-backend` `None` `ascend` A2, A3
`--prefill-attention-backend` `None` `ascend` A2, A3
`--decode-attention-backend` `None` `ascend` A2, A3
`--sampling-backend` `None` `pytorch`,
`ascend`		 A2, A3
`--grammar-backend` `None` `xgrammar`,
`outlines`,
`llguidance`		 A2, A3
`--radix-cache-backend` `None` Type: str A2, A3
`--mm-attention-backend` `None` `ascend_attn` A2, A3
`--dsa-prefill-backend` `flashmla_sparse` `flashmla_sparse`,
`flashmla_decode`,
`fa3`,
`tilelang`,
`aiter`		 Special for GPU
`--dsa-decode-backend` `fa3` `flashmla_prefill`,
`flashmla_kv`,
`fa3`,
`tilelang`,
`aiter`		 Special for GPU
`--fp8-gemm-backend` `auto` `auto`,
`deep_gemm`,
`flashinfer_trtllm`,
`flashinfer_cutlass`,
`flashinfer_deepgemm`,
`cutlass`,
`triton`,
`aiter`		 Special for GPU
`--disable-flashinfer-autotune` `False` bool flag
(set to enable)		 Special for GPU
## Speculative decoding
Argument Defaults Options Server supported
`--speculative-algorithm` `None` `EAGLE`,
`EAGLE3`,
`NEXTN`		 A2, A3
`--speculative-draft-model-path`
`--speculative-draft-model`		 `None` Type: str A2, A3
`--speculative-draft-model-revision` `None` Type: str,
`branch name`,
`tag name`,
`commit id`		 A2, A3
`--speculative-draft-load-format` `auto` `auto`,
`dummy`		 A2, A3
`--speculative-num-steps` `None` Type: int A2, A3
`--speculative-eagle-topk` `None` `1` (the only supported value on Ascend NPU) A2, A3
`--speculative-num-draft-tokens` `None` Type: int A2, A3
`--speculative-dflash-block-size` `None` Type: int A2, A3
`--speculative-accept-threshold-single` `1.0` Type: float Special for GPU
`--speculative-accept-threshold-acc` `1.0` Type: float Special for GPU
`--speculative-token-map` `None` Type: str; requires `--tp-size=1` A2, A3
`--speculative-attention-mode` `prefill` `prefill`,
`decode`		 A2, A3
`--speculative-moe-runner-backend` `None` `auto` A2, A3
`--speculative-moe-a2a-backend` `None` `ascend_fuseep` (the only supported value on Ascend NPU) A2, A3
`--speculative-draft-attention-backend` `None` `ascend` A2, A3
`--speculative-dflash-draft-window-size`
`--speculative-draft-window-size`		 `None` Type: int A2, A3
`--speculative-draft-model-quantization` `None` `unquant` (the only supported value for speculative decoding on Ascend NPU) A2, A3
## Ngram speculative decoding
Argument Defaults Options Server supported
`--speculative-ngram-min-match-window-size` `1` Type: int Experimental
`--speculative-ngram-max-match-window-size` `12` Type: int Experimental
`--speculative-ngram-min-bfs-breadth` `1` Type: int A2, A3
`--speculative-ngram-max-bfs-breadth` `10` Type: int A2, A3
`--speculative-ngram-match-type` `BFS` `BFS`,
`PROB`		 A2, A3. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion.
`--speculative-ngram-max-trie-depth` `18` Type: int A2, A3
`--speculative-ngram-capacity` `10000000` Type: int A2, A3
`--speculative-ngram-external-corpus-path` `None` Type: str A2, A3
`--speculative-ngram-external-sam-budget` `0` Type: int A2, A3
`--speculative-ngram-external-corpus-max-tokens` `10000000` Type: int A2, A3
`--speculative-adaptive` `False` bool flag (set to enable) A2, A3
`--speculative-adaptive-config` `None` Type: str A2, A3
`--speculative-skip-dp-mlp-sync` `False` bool flag (set to enable) A2, A3
## Expert parallelism
Argument Defaults Options Server supported
`--expert-parallel-size`
`--ep-size`
`--ep`		 `1` Type: int A2, A3
`--moe-a2a-backend` `none` `none`,
`deepep`,
`ascend_fuseep`(It is incompatible with eplb)		 A2, A3
`--moe-runner-backend` `auto` `auto`, `triton` A2, A3
`--flashinfer-mxfp4-moe-precision` `default` `default`,
`bf16`		 Special for GPU
`--enable-flashinfer-allreduce-fusion` `False` bool flag
(set to enable)		 Special for GPU
`--deepep-mode` `auto` `normal`,
`low_latency`,
`auto`		 A2, A3
`--fuseep-mode` `2` `1`,
`2`		 A2, A3
`--deepep-dispatcher-output-dtype` `auto` `auto`,
`bf16`,
`int8`
(When enabling DeepEP for a quantized model, set the dispatcher output dtype according to your model’s quantization. If the value is int8, you must also set the environment variable:DEEP\_NORMAL\_MODE\_USE\_INT8\_QUANT=1)		 A2, A3
`--deepep-config` `None` Type: str Special for GPU
`--ep-num-redundant-experts` `0` Type: int A2, A3
`--ep-dispatch-algorithm` `None` `static`,
`dynamic`,
`fake`		 A2, A3
`--init-expert-location` `trivial` `trivial`,
``,
``,
``		 A2, A3
`--enable-eplb` `False` bool flag
(set to enable)		 A2, A3
`--eplb-algorithm` `deepseek` `auto`,
`deepseek`		 A2, A3
`--eplb-rebalance-num-iterations` `1000` Type: int A2, A3
`--eplb-rebalance-layers-per-chunk` `None` Type: int A2, A3
`--eplb-min-rebalancing-utilization-threshold` `1.0` Type: float A2, A3
`--expert-distribution-recorder-mode` `None` `stat`,
`stat_approx`,
`per_pass`,
`per_token`		 A2, A3
`--expert-distribution-recorder-buffer-size` `None` Type: int A2, A3
`--expert-balancedness-report-mode` off off, server\_log, prometheus, both A2, A3
`--moe-dense-tp-size` `None` `1` A2, A3
`--elastic-ep-backend` `None` `none`, `mooncake` Special for GPU
`--mooncake-ib-device` `None` Type: str Special for GPU
`--enable-waterfill` `False` bool flag (set to enable) A2, A3
## Mamba cache
Argument Defaults Options Server supported
`--max-mamba-cache-size` `None` Type: int A2, A3
`--mamba-ssm-dtype` `float32` `float32`,
`bfloat16`,
`float16`		 A2, A3
`--mamba-full-memory-ratio` `0.9` Type: float A2, A3
`--mamba-radix-cache-strategy` `auto` `auto`,
`no_buffer`,
`extra_buffer`		 A2, A3
`--mamba-track-interval` `256` Type: int A2, A3
## Hierarchical cache
Argument Defaults Options Server supported
`--enable-hierarchical-cache` `False` bool flag
(set to enable).
Currently, mamba cache is not supported.		 A2, A3
`--hicache-ratio` `2.0` Type: float A2, A3
`--hicache-size` `0` Type: int A2, A3
`--hicache-write-policy` `write_through` `write_back`,
`write_through`,
`write_through_selective`		 A2, A3
`--hicache-io-backend` `kernel` `kernel_ascend`,
`direct`		 A2, A3
`--hicache-mem-layout` `layer_first` `page_first_direct`,
`page_first_kv_split`		 A2, A3
`--hicache-storage-backend` `None` `file` A2, A3
`--hicache-storage-prefetch-policy` `timeout` `best_effort`,
`wait_complete`,
`timeout`		 Special for GPU
`--hicache-storage-backend-extra-config` `None` Type: str Special for GPU
## LMCache
Argument Defaults Options Server supported
`--enable-lmcache` `False` bool flag
(set to enable)		 Special for GPU
`--lmcache-config-file` `None` Type: str Special for GPU
## Diffusion LLM
Argument Defaults Options Server supported
`--dllm-algorithm` `None` Type: str A2, A3
`--dllm-algorithm-config` `None` Type: str A2, A3
## Offloading (must be used with `--disable-cuda-graph`)
Argument Defaults Options Server supported
`--cpu-offload-gb` `0` Type: int A2, A3
`--offload-group-size` `-1` Type: int (DeepSeek only) A2, A3
`--offload-num-in-group` `1` Type: int (DeepSeek only) A2, A3
`--offload-prefetch-step` `1` Type: int (DeepSeek only) A2, A3
`--offload-mode` `cpu` `cpu` (DeepSeek only)
`meta` (DeepSeek only)
`sharded_gpu` (DeepSeek only, only support tp=1 dp>1)		 A2, A3
## Optimization/debug options
Argument Defaults Options Server supported
`--disable-radix-cache` `False` bool flag
(set to enable)		 A2, A3
`--cuda-graph-config` `None` Type: JSON A2, A3
`--cuda-graph-backend-decode` `None` `full`,
`disabled`		 A2, A3
`--cuda-graph-backend-prefill` `None` `disabled` A2, A3
`--cuda-graph-max-bs-decode` `None` Type: int A2, A3
`--cuda-graph-max-bs-prefill` `None` Type: int A2, A3
`--cuda-graph-bs-decode` `None` Type: List\[int] A2, A3
`--cuda-graph-bs-prefill` `None` Type: List\[int] A2, A3
`--disable-prefill-cuda-graph` `False` bool flag
(set to enable)		 A2, A3
`--disable-decode-cuda-graph` `False` bool flag
(set to enable)		 A2, A3
`--cuda-graph-bs` `None` List\[int] A2, A3
`--disable-cuda-graph` `False` bool flag
(set to enable)		 A2, A3
`--disable-cuda-graph-padding` `False` bool flag
(set to enable)		 A2, A3
`--enable-profile-cuda-graph` `False` bool flag
(set to enable)		 A2, A3
`--enable-cudagraph-gc` `False` bool flag
(set to enable)		 A2, A3
`--enable-nccl-nvls` `False` bool flag
(set to enable)		 Special for GPU
`--enable-symm-mem` `False` bool flag
(set to enable)		 Special for GPU
`--disable-flashinfer-cutlass-moe-fp4-allgather` `False` bool flag
(set to enable)		 Special for GPU
`--enable-tokenizer-batch-encode` `False` bool flag
(set to enable)		 A2, A3
`--disable-tokenizer-batch-decode` `False` bool flag
(set to enable)		 A2, A3
`--disable-custom-all-reduce` `False` bool flag
(set to enable)		 A2, A3
`--enable-mscclpp` `False` bool flag
(set to enable)		 Special for GPU
`--pre-warm-nccl` `False` bool flag (set to enable) A2, A3
`--enable-torch-symm-mem` `False` bool flag
(set to enable)		 Special for GPU
`--disable-overlap-schedule` `False` bool flag
(set to enable)		 A2, A3
`--enable-mixed-chunk` `False` bool flag
(set to enable)		 A2, A3
`--enable-dp-attention` `False` bool flag
(set to enable)		 A2, A3
`--enable-dp-attention-local-control-broadcast` `False` bool flag
(set to enable)		 A2, A3
`--enable-dp-lm-head` `False` bool flag
(set to enable)		 A2, A3
`--enable-two-batch-overlap` `False` bool flag
(set to enable)		 Planned
`--enable-single-batch-overlap` `False` bool flag
(set to enable)		 Special for GPU
`--tbo-token-distribution-threshold` `0.48` Type: float Planned
`--enable-torch-compile` `False` bool flag
(set to enable)		 A2, A3
`--enable-torch-compile-debug-mode` `False` bool flag
(set to enable)		 A2, A3
`--enforce-piecewise-cuda-graph` `False` bool flag
(set to enable);
Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported.		 A2, A3
`--piecewise-cuda-graph-tokens` `None` Type: JSON
list		 A2, A3
`--piecewise-cuda-graph-compiler` `eager` `eager` A2, A3
`--torch-compile-max-bs` `32` Type: int A2, A3
`--piecewise-cuda-graph-max-tokens` `None` Type: int A2, A3
`--enable-p2p-check` `False` bool flag
(set to enable)		 Special for GPU
`--triton-attention-reduce-in-fp32` `False` bool flag
(set to enable)		 Special for GPU
`--triton-attention-num-kv-splits` `8` Type: int Special for GPU
`--triton-attention-split-tile-size` `None` Type: int Special for GPU
`--delete-ckpt-after-loading` `False` bool flag
(set to enable)		 A2, A3
`--enable-memory-saver` `False` bool flag
(set to enable)		 A2, A3
`--enable-weights-cpu-backup` `False` bool flag
(set to enable)		 A2, A3
`--enable-draft-weights-cpu-backup` `False` bool flag
(set to enable)		 A2, A3
`--allow-auto-truncate` `False` bool flag
(set to enable)		 A2, A3
`--enable-custom-logit-processor` `False` bool flag
(set to enable)		 A2, A3
`--flashinfer-mla-disable-ragged` `False` bool flag
(set to enable)		 Special for GPU
`--disable-shared-experts-fusion` `True` bool flag
(set to enable)		 A2, A3
`--enforce-shared-experts-fusion` `False` bool flag
(set to enable)		 A2, A3
`--disable-chunked-prefix-cache` `True` bool flag
(set to enable)		 A2, A3
`--image-processor-backend` `auto` `auto`, `torchvision`, `pil` A2, A3
`--disable-fast-image-processor` `False` bool flag
(set to enable)		 A2, A3
`--keep-mm-feature-on-device` `False` bool flag
(set to enable)		 A2, A3
`--enable-return-hidden-states` `False` bool flag
(set to enable)		 A2, A3
`--enable-return-routed-experts` `False` bool flag
(set to enable)		 A2, A3
`--scheduler-recv-interval` `1` Type: int A2, A3
`--numa-node` `None` List\[int] A2, A3
`--enable-deterministic-inference` `False` bool flag
(set to enable)		 A2, A3
`--rl-on-policy-target` `None` `fsdp` A2, A3
`--disable-attn-tp-gather` `False` bool flag (set to enable) A2, A3
`--enable-layerwise-nvtx-marker` `False` bool flag
(set to enable)		 Special for GPU
`--enable-attn-tp-input-scattered` `False` bool flag
(set to enable)		 A2, A3
`--enable-prefill-cp` `False` bool flag (set to enable) A2, A3
`--cp-strategy` `None` `zigzag` A2, A3
`--enable-fused-qk-norm-rope` `False` bool flag
(set to enable)		 Special for GPU
`--enable-precise-embedding-interpolation` `False` bool flag
(set to enable)		 A2, A3
`--gc-threshold` `None` List\[int] A2, A3
## Dynamic batch tokenizer
Argument Defaults Options Server supported
`--enable-dynamic-batch-tokenizer` `False` bool flag
(set to enable)		 A2, A3
`--dynamic-batch-tokenizer-batch-size` `32` Type: int A2, A3
`--dynamic-batch-tokenizer-batch-timeout` `0.002` Type: float A2, A3
## Debug tensor dumps
Argument Defaults Options Server supported
`--debug-tensor-dump-output-folder` `None` Type: str A2, A3
`--debug-tensor-dump-layers` `None` List\[int] A2, A3
`--debug-tensor-dump-input-file` `None` Type: str A2, A3
## PD disaggregation
Argument Defaults Options Server supported
`--disaggregation-mode` `null` `null`,
`prefill`,
`decode`		 A2, A3
`--disaggregation-transfer-backend` `mooncake` `ascend` (default `mooncake` not supported on NPU, must be specified manually) A2, A3
`--disaggregation-bootstrap-port` `8998` Type: int A2, A3
`--disaggregation-ib-device` `None` Type: str Special for GPU
`--disaggregation-decode-enable-offload-kvcache` `False` bool flag
(set to enable)		 A2, A3
`--num-reserved-decode-tokens` `512` Type: int A2, A3
`--disaggregation-decode-polling-interval` `1` Type: int A2, A3
`--optimistic-prefill-attempts` `0` Type: int A2, A3
## Encode prefill disaggregation
Argument Defaults Options Server supported
`--enable-adaptive-dispatch-to-encoder` `False` bool flag
(set to enable adaptive dispatch)		 A2, A3
`--encoder-only` `False` bool flag
(set to launch an encoder-only server)		 A2, A3
`--language-only` `False` bool flag
(set to load weights for the language model only)		 A2, A3
`--encoder-transfer-backend` `zmq_to_scheduler` `zmq_to_scheduler`,
`zmq_to_tokenizer`,
`mooncake`		 A2, A3
`--encoder-urls` `[]` List\[str]
(List of encoder server urls)		 A2, A3
`--encoder-bootstrap-port` `8997` Type: int A2, A3
`--encoder-register-urls` `[]` List\[str] A2, A3
## Custom weight loader
Argument Defaults Options Server supported
`--custom-weight-loader` `None` List\[str] A2, A3
`--weight-loader-disable-mmap` `False` bool flag
(set to enable)		 A2, A3
`--weight-loader-prefetch-checkpoints` `False` bool flag
(set to enable)		 A2, A3
`--weight-loader-prefetch-num-threads` `4` Type: int A2, A3
`--remote-instance-weight-loader-seed-instance-ip` `None` Type: str A2, A3
`--remote-instance-weight-loader-seed-instance-service-port` `None` Type: int A2, A3
`--remote-instance-weight-loader-send-weights-group-ports` `None` Type: JSON
list		 A2, A3
`--remote-instance-weight-loader-backend` `nccl` `transfer_engine`,
`nccl`,
`modelexpress`		 A2, A3
`--weight-loader-drop-cache-after-load` `False` bool flag (set to enable) A2, A3
`--remote-instance-weight-loader-start-seed-via-transfer-engine` `False` bool flag
(set to enable)		 Special for GPU
## For PD-multiplexing
Argument Defaults Options Server supported
`--enable-pdmux` `False` bool flag
(set to enable)		 Special for GPU
`--pdmux-config-path` `None` Type: str Special for GPU
`--sm-group-num` `8` Type: int Special for GPU
## For multi-modal
Argument Defaults Options Server supported
`--enable-broadcast-mm-inputs-process` `False` bool flag
(set to enable)		 A2, A3
`--mm-process-config` `None` Type: JSON / Dict A2, A3
`--mm-enable-dp-encoder` `False` bool flag
(set to enable)		 A2, A3
`--mm-processor-worker-num` `0` Type: int A2, A3
`--mm-io-worker-num` `0` Type: int A2, A3
`--limit-mm-data-per-request` `None` Type: JSON / Dict A2, A3
## For checkpoint decryption
Argument Defaults Options Server supported
`--decrypted-config-file` `None` Type: str A2, A3
`--decrypted-draft-config-file` `None` Type: str A2, A3
`--enable-prefix-mm-cache` `False` bool flag
(set to enable)		 A2, A3
## Forward hooks
Argument Defaults Options Server supported
`--forward-hooks` `None` Type: JSON list A2, A3
`--enable-quant-communications` `False` bool flag (set to enable)
(When using DeepEP with a quantized model, you must set --deepep-dispatcher-output-dtype)		 A2, A3
## msProbe
Argument Defaults Options Server supported
`--msprobe-dump-config` `None` Type: str A2, A3
## Configuration file support
Argument Defaults Options Server supported
`--config` `None` Type: str A2, A3
## Other params The following parameters are not supported because the required third-party components (e.g., Ktransformer, checkpoint-engine) are not compatible with the NPU.
Argument Defaults Options
`--checkpoint-engine-wait-weights-before-ready` `False` bool flag (set to enable)
`--kt-weight-path` `None` Type: str
`--kt-method` `AMXINT4` Type: str
`--kt-cpuinfer` `None` Type: int
`--kt-threadpool-count` 2 Type: int
`--kt-num-gpu-experts` `None` Type: int
`--kt-max-deferred-experts-per-token` `None` Type: int
The following parameters have known functional deficiencies in the upstream community codebase
Argument Defaults Options
`--tool-server` `None` Type: str
# Supported Models Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_models This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language Models, Diffusion Language Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included. You are welcome to enable various models based on your business requirements. ## Large Language Models
Models Model Family A2 Supported A3 Supported
Eco-Tech/DeepSeek-V4-Pro-w4a8-mtp DeepSeek ✅ ✅
Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp DeepSeek ✅ ✅
sgl-npu/DeepSeek-V3.1-w8a8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-V3.2-W8A8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-R1-0528-W8A8 DeepSeek ✅ ✅
sgl-npu/DeepSeek-V2-Lite-W8A8 DeepSeek ✅ ✅
Qwen/Qwen3.8-2.4T-A95B Qwen3.8 ✅ ✅
Qwen/Qwen3.6-35B-A3B Qwen3.6 ✅ ✅
Eco-Tech/Qwen3.6-27B-w8a8 Qwen3.6 ✅ ✅
Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp Qwen3.5 ✅ ✅
Eco-Tech/Qwen3.5-27B-w8a8-mtp Qwen3.5 ✅ ✅
Qwen/Qwen3.5-9B Qwen3.5 ✅ ✅
Qwen/Qwen3.5-4B Qwen3.5 ✅ ✅
Qwen/Qwen3.5-0.8B Qwen3.5 ✅ ✅
Qwen/Qwen3-30B-A3B-Instruct-2507 Qwen3 ✅ ✅
Qwen/Qwen3-32B Qwen3 ✅ ✅
Qwen/Qwen3-0.6B Qwen3 ✅ ✅
sgl-npu/Qwen3-235B-A22B-W8A8 Qwen3 ✅ ✅
Qwen/Qwen3-Next-80B-A3B-Instruct Qwen3 ✅ ✅
Eco-Tech/Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot Qwen3 ✅ ✅
Qwen/Qwen2.5-7B-Instruct Qwen2.5 ✅ ✅
sgl-npu/QwQ-32B-W8A8 QWQ ✅ ✅
LLM-Research/Llama-4-Scout-17B-16E-Instruct Llama ✅ ✅
AI-ModelScope/Llama-3.1-8B-Instruct Llama ✅ ✅
LLM-Research/llama-2-7b Llama ✅ ✅
LLM-Research/Llama-3.2-1B-Instruct Llama ✅ ✅
mistralai/Mistral-7B-Instruct-v0.2 Mistral ✅ ✅
google/gemma-3-4b-it Gemma ✅ ✅
microsoft/Phi-4-multimodal-instruct Phi ✅ ✅
allenai/OLMoE-1B-7B-0924 OLMoE ✅ ✅
stabilityai/stablelm-2-1\_6b StableLM ✅ ✅
CohereForAI/c4ai-command-r-v01 Command-R ✅ ✅
ZhipuAI/chatglm2-6b ChatGLM ✅ ✅
LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct ExaONE 3 ✅ ✅
xverse/XVERSE-MoE-A36B XVERSE ✅ ✅
HuggingFaceTB/SmolLM-1.7B SmolLM ✅ ✅
Eco-Tech/GLM-5.2-w8a8 GLM-5 ✅ ✅
Eco-Tech/GLM-5.1-w4a8 GLM-5 ✅ ✅
Eco-Tech/GLM-5-w4a8 GLM-5 ✅ ✅
ZhipuAI/glm-4-9b-chat GLM-4 ✅ ✅
iridiumine/MiMo-V2-Flash-W8A8 MiMo ✅ ✅
XiaomiMiMo/MiMo-7B-RL MiMo ✅ ✅
arcee-ai/AFM-4.5B-Base Arcee AFM-4.5B ✅ ✅
Howeee/persimmon-8b-chat Persimmon ✅ ✅
inclusionAI/Ling-lite Ling ✅ ✅
ibm-granite/granite-3.1-8b-instruct Granite ✅ ✅
AI-ModelScope/dbrx-instruct DBRX (Databricks) ✅ ✅
baichuan-inc/Baichuan2-13B-Chat Baichuan 2 (7B, 13B) ✅ ✅
PaddlePaddle/ERNIE-4.5-21B-A3B-PT ERNIE-4.5 (4.5, 4.5MoE series) ✅ ✅
OpenBMB/MiniCPM3-4B MiniCPM (v3, 4B) ✅ ✅
sgl-npu/Kimi-K3-W4A8 Kimi ✅ ✅
Eco-Tech/Kimi-K2.6-w4a8 Kimi ✅ ✅
Eco-Tech/Kimi-K2.5-w4a8 Kimi ✅ ✅
moonshotai/Kimi-K2-Thinking Kimi ✅ ✅
eigen-ai-labs/gpt-oss-120b-bf16 GPTOSS ✅ ✅
allenai/OLMo-2-1124-7B-Instruct OLMo ✅ ✅
Eco-Tech/MiniMax-M2.5-w8a8-QuaRot MiniMax-M2.5 ✅ ✅
cyankiwi/MiniMax-M2-BF16 MiniMax-M2 ✅ ✅
upstage/SOLAR-10.7B-Instruct-v1.0 Solar ✅ ✅
FLM/Tele-FLM Tele FLM (52B-1T) ✅ ✅
bigcode/starcoder2-7b StarCoder2 ✅ ✅
arcee-ai/Trinity-Mini Trinity (Nano, Mini) ✅ ✅
OrionStarAI/Orion-14B-Base Orion (14B) ✅ ✅
EleutherAI/gpt-j-6b GPT-J (6B) ✅ ✅
LiquidAI/LFM2.5-1.2B-Instruct LFM2 (230M, 350M, 1.2B) ✅ ✅
LiquidAI/LFM2-8B-A1B LFM2-MoE (8B-A1B, 24B-A2B) ✅ ✅
tiiuae/Falcon-H1-34B-Instruct Falcon-H1 (0.5B-34B) ✅ ✅
ibm-granite/granite-4.0-h-micro IBM Granite 4.0 (Hybrid, Dense) ✅ ✅
ibm-granite/granite-4.0-micro IBM Granite 4.0 (Hybrid, Dense) ✅ ✅
poolside/Laguna-XS.2 Laguna XS.2 (poolside) ✅ ✅
Tencent-Hunyuan/Hy3 Hunyuan ✅ ✅
## Multimodal Language Models
Models Model Family (Variants) A2 Supported A3 Supported
Qwen/Qwen2.5-VL-3B-Instruct Qwen2.5-VL ✅ ✅
Qwen/Qwen2.5-VL-72B-Instruct Qwen2.5-VL ✅ ✅
Qwen/Qwen3-VL-30B-A3B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-8B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-4B-Instruct Qwen3-VL ✅ ✅
Qwen/Qwen3-VL-235B-A22B-Instruct Qwen3-VL ✅ ✅
deepseek-ai/deepseek-vl2 DeepSeek-VL2 ✅ ✅
deepseek-ai/Janus-Pro-1B Janus-Pro (1B, 7B) ✅ ✅
deepseek-ai/Janus-Pro-7B Janus-Pro (1B, 7B) ✅ ✅
OpenBMB/MiniCPM-V-2\_6 MiniCPM-V / MiniCPM-o ✅ ✅
OpenBMB/MiniCPM-o-2\_6 MiniCPM-V / MiniCPM-o ✅ ✅
google/gemma-3-4b-it Gemma 3 (Multimodal) ✅ ✅
mistralai/Mistral-Small-3.1-24B-Instruct-2503 Mistral-Small-3.1-24B ✅ ✅
microsoft/Phi-4-multimodal-instruct Phi-4-multimodal-instruct ✅ ✅
XiaomiMiMo/MiMo-VL-7B-RL MiMo-VL (7B) ✅ ✅
AI-ModelScope/llava-v1.6-34b LLaVA (v1.5 & v1.6) ✅ ✅
lmms-lab/llava-next-72b LLaVA-NeXT (8B, 72B) ✅ ✅
moonshotai/Kimi-VL-A3B-Instruct Kimi-VL (A3B) ✅ ✅
ZhipuAI/GLM-4.5V GLM-4.5V (106B) ✅ ✅
LLM-Research/Llama-3.2-11B-Vision-Instruct Llama 3.2 Vision (11B) ✅ ✅
rednote-hilab/dots.ocr DotsVLM-OCR ✅ ✅
Qwen/Qwen3-Omni-30B-A3B-Instruct Qwen3-Omni ✅ ✅
stepfun-ai/Step3-VL-10B Step3-VL (10B) ✅ ✅
deepseek-ai/DeepSeek-OCR-2 DeepSeek-OCR / OCR-2 ✅ ✅
lmms-lab/LLaVA-NeXT-Video-7B LLaVA video variants (LLaVA-NeXT-Video, LLaVA-OneVision) ✅ ✅
ZhipuAI/GLM-OCR GLM-OCR ✅ ✅
AI-ModelScope/whisper-large-v3 Whisper ✅ ✅
Qwen/Qwen3-ASR-1.7B Qwen3-ASR (0.6B, 1.7B) ✅ ✅
LiquidAI/LFM2.5-VL-1.6B LFM2-VL ✅ ✅
## Embedding Models
Models Model Family A2 Supported A3 Supported
intfloat/e5-mistral-7b-instruct E5 (Llama/Mistral based) ✅ ✅
iic/gte-Qwen2-1.5B-instruct GTE-Qwen2 ✅ ✅
Qwen/Qwen3-Embedding-8B Qwen3-Embedding ✅ ✅
iic/gme-Qwen2-VL-2B-Instruct GME (Multimodal) ✅ ✅
AI-ModelScope/clip-vit-large-patch14-336 CLIP ✅ ✅
BAAI/bge-large-en-v1.5 BGE ✅ ✅
## Reward Models
Models Model Family A2 Supported A3 Supported
Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 Llama3.1 Reward ✅ ✅
Shanghai\_AI\_Laboratory/internlm2-7b-reward InternLM 2 Reward ✅ ✅
Qwen/Qwen2.5-Math-RM-72B Qwen2.5 Reward - Math ✅ ✅
Howeee/Qwen2.5-1.5B-apeach Qwen2.5 Reward - Sequence ✅ ✅
## Rerank Models
Models Model Family A2 Supported A3 Supported
BAAI/bge-reranker-v2-m3 BGE-Reranker ✅ ✅
Qwen/Qwen3-Reranker-8B Qwen3-Reranker (decoder-only yes/no) ✅ ✅
Qwen/Qwen3-VL-Reranker-2B Qwen3-VL-Reranker (multimodal yes/no) ✅ ✅
# CPU Servers Source: https://docs.sglang.io/docs/hardware-platforms/cpu_server The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers. SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions, which are 4th generation or newer Intel® Xeon® Scalable Processors. A number of popular LLMs are optimized and run efficiently on CPU, including the most notable open-source models like Llama series, Qwen series, and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus. Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found. ## Installation ### Install Using Docker It is recommended to use Docker for setting up the SGLang environment. #### Pull from Docker Hub Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository. The [CPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xeon) end with `xeon` suffix. The image pulling command is like: ```bash Command theme={null} docker pull lmsysorg/sglang:v0.5.13-xeon ``` #### Build from Dockerfile A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation from latest source code. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t sglang-cpu:latest -f xeon.Dockerfile . # Initiate a docker container docker run \ -it \ --privileged \ --ipc=host \ --network=host \ -v /dev/shm:/dev/shm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ -e "HF_TOKEN=" \ sglang-cpu:latest /bin/bash ``` ### Install From Source If you prefer to install SGLang in a bare metal environment, the setup process is as follows: Please install the required packages and libraries beforehand if they are not already present on your system. You can refer to the Ubuntu-based installation commands in [the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L7) for guidance. 1. Install `uv` package manager, then create and activate a virtual environment: ```bash Command theme={null} # Taking '/opt' as the example uv env folder, feel free to change it as needed cd /opt curl -LsSf https://astral.sh/uv/install.sh | sh source $HOME/.local/bin/env uv venv --python 3.12 source .venv/bin/activate ``` 2. Create a config file to direct the installation channel (a.k.a. index-url) of `torch` related packages: ```bash Command theme={null} vim .venv/uv.toml ``` Press 'a' to enter insert mode of `vim`, paste the following content into the created file ```file theme={null} [[index]] name = "torch" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "torchvision" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "torchaudio" url = "https://download.pytorch.org/whl/cpu" [[index]] name = "triton" url = "https://download.pytorch.org/whl/cpu" ``` Save the file (in `vim`, press 'esc' to exit insert mode, then ':x+Enter'), and set it as the default `uv` config. ```bash Command theme={null} export UV_CONFIG_FILE=/opt/.venv/uv.toml ``` 3. Clone the `sglang` source code and build the packages ```bash Command theme={null} # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git cd sglang git checkout # Use dedicated toml file cd python cp pyproject_cpu.toml pyproject.toml # Install SGLang dependent libs, and build SGLang main package uv pip install --upgrade pip setuptools uv pip install . # Build the CPU backend kernels cd sglang/kernels/aot cp pyproject_cpu.toml pyproject.toml uv pip install . ``` 4. Set the required environment variables ```bash Command theme={null} export SGLANG_USE_CPU_ENGINE=1 # Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2 ``` Notes: * Note that the environment variable `SGLANG_USE_CPU_ENGINE=1` is required to enable the SGLang service with the CPU engine. * If you encounter code compilation issues during the `sgl-kernel` building process, please check your `gcc` and `g++` versions and upgrade them if they are outdated. It is recommended to use `gcc-13` and `g++-13` as they have been verified in the official Docker container. * The system library path is typically located in one of the following directories: `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/` and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu` is used. Please adjust the path according to your server configuration. * It is recommended to add the following to your `~/.bashrc` file to avoid setting these variables every time you open a new terminal: ```bash Command theme={null} source .venv/bin/activate export SGLANG_USE_CPU_ENGINE=1 export LD_LIBRARY_PATH= export LD_PRELOAD= ``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash Launch Server theme={null} sglang serve \ --model-path \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --host 0.0.0.0 \ --tp 6 ``` Notes: 1. For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`. 2. The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6). The number of TP specified is how many TP ranks will be used during the execution. On a CPU platform, a TP rank means a sub-NUMA cluster (SNC). Usually we can get the SNC information (How many available) from the Operating System with e.g. `lscpu` command. If the specified TP rank number differs from the total SNC count, the system will automatically utilize the first `n` SNCs. Note that `n` cannot exceed the total SNC number, doing so will result in an error. `SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank. **example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server, which has 43-43-42 cores on the 3 SNCs of a socket, we should set: ```bash Command theme={null} export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253" ``` This configuration is equivalent to: * rank 0: `numactl -C 0-39 -m 0` * rank 1: `numactl -C 43-82 -m 1` * rank 2: `numactl -C 86-125 -m 2` * rank 3: `numactl -C 128-167 -m 3` * rank 4: `numactl -C 171-210 -m 4` * rank 5: `numactl -C 214-253 -m 5` **example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server, which has 32-32-32 cores on the 3 SNCs in a socket, we should set: ```bash Command theme={null} export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191" ``` This configuration is equivalent to: * rank 0: `numactl -C 0-95 -m 0-2` * rank 1: `numactl -C 96-191 -m 3-5` Please beware that with SGLANG\_CPU\_OMP\_THREADS\_BIND set, the available memory amounts of the ranks may not be determined in prior. You may need to set proper `--max-total-tokens` to avoid the out-of-memory error. 3. For optimizing decoding with torch.compile, please add the flag `--enable-torch-compile`. To specify the maximum batch size when using `torch.compile`, set the flag `--torch-compile-max-bs`. For example, `--enable-torch-compile --torch-compile-max-bs 4` means using `torch.compile` and setting the maximum batch size to 4. 4. A warmup step is automatically triggered when the service is started. The server is ready when you see the log `The server is fired up and ready to roll!`. ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. Run the command in another terminal. An example command would be: ```bash Run Benchmark theme={null} python -m sglang.bench_serving \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1 \ --request-rate inf \ --random-range-ratio 1.0 ``` Detailed parameter descriptions are available via the command: ```bash Benchmark Help theme={null} python -m sglang.bench_serving -h ``` Additionally, requests can be formatted using [the OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g., using `curl`) or through your own scripts. ## Example Usage Commands Large Language Models can range from fewer than 1 billion to several hundred billion parameters. Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer, or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common 4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors. ### Example: Running DeepSeek-V3.1-Terminus An example command to launch service of W8A8\_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server: ```bash W8A8_INT8 theme={null} sglang serve \ --model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --quantization w8a8_int8 \ --enable-torch-compile \ --torch-compile-max-bs 4 \ --host 0.0.0.0 \ --tp 6 ``` Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be: ```bash FP8 theme={null} sglang serve \ --model-path deepseek-ai/DeepSeek-V3.1-Terminus \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --enable-torch-compile \ --torch-compile-max-bs 4 \ --host 0.0.0.0 \ --tp 6 ``` Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment. The value `4` in the examples is illustrative. ### Example: Running Llama-3.2-3B An example command to launch service of Llama-3.2-3B with BF16 precision: ```bash BF16 theme={null} sglang serve \ --model-path meta-llama/Llama-3.2-3B-Instruct \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --enable-torch-compile \ --torch-compile-max-bs 16 \ --host 0.0.0.0 \ --tp 3 ``` The example command to launch service of W8A8\_INT8 version of Llama-3.2-3B: ```bash W8A8_INT8 theme={null} sglang serve \ --model-path RedHatAI/Llama-3.2-3B-quantized.w8a8 \ --trust-remote-code \ --disable-overlap-schedule \ --device cpu \ --quantization w8a8_int8 \ --enable-torch-compile \ --torch-compile-max-bs 16 \ --host 0.0.0.0 \ --tp 3 ``` Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server. Once the server have been launched, you can test it using the `bench_serving` command or create your own commands or scripts following [the benchmarking example](#benchmarking-with-requests). # Moore Threads GPUs Source: https://docs.sglang.io/docs/hardware-platforms/mthreads_gpu This document describes how run SGLang on Moore Threads GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). ## Install SGLang You can install SGLang using one of the methods below. ### Install from Source ```bash theme={null} # Use the default branch git clone https://github.com/sgl-project/sglang.git cd sglang # Compile sgl-kernel pip install --upgrade pip cd python/sglang/kernels/aot python setup_musa.py install # Install sglang python package along with diffusion support cd ../../../.. rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml pip install -e "python[all_musa]" ``` # NVIDIA GPUs Source: https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus Please refer to the [Installation Guide](../get-started/install) to get started with SGLang on NVIDIA GPUs. # NVIDIA Jetson Orin Source: https://docs.sglang.io/docs/hardware-platforms/nvidia_jetson Guide for installing and running SGLang on NVIDIA Jetson Orin devices. ## Prerequisites Before starting, ensure the following: * [**NVIDIA Jetson AGX Orin Devkit**](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with **JetPack 6.1** or later. * **CUDA Toolkit** and **cuDNN** are installed. * Verify that the Jetson AGX Orin is in **high-performance mode**: ```bash theme={null} sudo nvpmodel -m 0 ``` *** ## Installing and running SGLang with Jetson Containers Clone the jetson-containers github repository: ```bash theme={null} git clone https://github.com/dusty-nv/jetson-containers.git ``` Run the installation script: ```bash theme={null} bash jetson-containers/install.sh ``` Build the container image: ```bash theme={null} jetson-containers build sglang ``` Run the container: ``` jetson-containers run $(autotag sglang) ``` Or you can also manually run a container with this command: ``` docker run --runtime nvidia -it --rm --network=host IMAGE_NAME ``` *** ## Running Inference Launch the server: ```bash theme={null} python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \ --device cuda \ --dtype half \ --attention-backend flashinfer \ --mem-fraction-static 0.8 \ --context-length 8192 ``` The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments). After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#usage) to test the usability. *** ## Structured output with XGrammar Please refer to [SGLang doc structured output](../advanced_features/structured_outputs). *** Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez). ## References * [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin) # Hardware Platforms Source: https://docs.sglang.io/docs/hardware-platforms/overview Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CPUs, and more. * [NVIDIA GPUs](./nvidia-gpus) * [AMD GPUs](./amd_gpu) * [Ascend NPUs](./ascend-npus/getting-started/installation) * [CPU Server](./cpu_server) * [NVIDIA Jetson Orin](./nvidia_jetson) * [TPU](./tpu) * [XPU](./xpu) # SGLang Plugin System Source: https://docs.sglang.io/docs/hardware-platforms/plugin ## Overview Allows hardware vendors and developers to extend SGLang **without modifying the main repository code**. The framework provides two plugin types, both discovered via Python's standard `setuptools` entry\_points:
Plugin Type Entry Point Group Purpose
Hardware Platform Plugin sglang.srt.platforms Register a custom hardware platform (device operations, KV cache pools, attention backends, graph capture, compilation backends, etc.)
General Plugin sglang.srt.plugins Inject hooks (before/after/around/replace) into any function/method, or replace entire classes
### Principles * **Non-intrusive**: Existing CUDA/ROCm/NPU/XPU code remains unchanged. OOT code paths are added alongside existing hardware-specific logic. * **Zero configuration**: Plugins are automatically discovered after `pip install`, no sglang code changes required. * **Environment variable control**: `SGLANG_PLATFORM` selects or validates the active platform plugin; `SGLANG_PLUGINS` (comma-separated) controls which general plugins to load. ### Current Scope & Future Direction The plugin system currently targets **out-of-tree (OOT) hardware platforms** — enabling new devices to integrate with SGLang without any changes to the main repository. The main-repo hardware paths (CUDA, ROCm, NPU, XPU, etc.) continue to use the existing `is_cuda()`/`is_npu()`/… utility functions. As the plugin interfaces mature and stabilize, in-tree hardware backends can be gradually migrated to the same plugin architecture. This would replace the scattered `if device == "cuda" … elif device == "npu" …` branches throughout the codebase with a single polymorphic dispatch through the platform interface, making each hardware backend self-contained and the core engine hardware-agnostic. ## Architecture ### Platform Hierarchy The platform hierarchy uses a DeviceMixin pattern to share device operations between SRT (LLM inference) and Multimodal subsystems: ``` DeviceMixin (shared device identity + operations) ├── SRTPlatform(DeviceMixin) # + graph runner, KV pool, … │ └── MySRTPlatform(SRTPlatform, MyDeviceMixin) # OOT plugin └── MMPlatform(DeviceMixin) # + attention backend, VAE, … (future) └── MyMMPlatform(MMPlatform, MyDeviceMixin) # OOT plugin ``` Key design points: * **DeviceMixin** provides platform identity queries (`is_cuda()`, `is_npu()`, etc.) and device operations (`set_device()`, `get_device_name()`, etc.) * **SRTPlatform** adds SRT-specific factory methods, capability flags, and lifecycle hooks * OOT plugins implement a **device mixin** (vendor-specific operations) and compose it with **SRTPlatform** via multiple inheritance * All methods are **instance methods** (not classmethods), called through the `current_platform` singleton * Device operations and factory methods raise `NotImplementedError` by default (fail-fast) * Capability flags use safe conservative defaults (`False`/`pass`) * Methods are annotated `[Active]` (called by SGLang core) or `[Planned]` (reserved for future migration) ### Platform Discovery (`current_platform`) `current_platform` is a **lazy singleton** in `sglang.srt.platforms`. On first access it resolves the active platform through the following priority chain: ``` entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadata only) │ ├─ SGLANG_PLATFORM set (front-loading filter): │ ├─ Name not found in discovered → RuntimeError │ ├─ activate() returns non-None → load that platform │ └─ activate() returns None → RuntimeError (hardware unavailable) │ └─ SGLANG_PLATFORM unset (auto-discover, activate all): ├─ 0 activated + SGLANG_USE_CPU_ENGINE=1 → fallback CpuSRTPlatform ├─ 0 activated + CUDA available → fallback CudaSRTPlatform ├─ 0 activated + ROCm available → fallback RocmSRTPlatform ├─ 0 activated + XPU available → fallback XpuSRTPlatform ├─ 0 activated + none of the above → fallback base SRTPlatform ├─ 1 activated → use it └─ N activated → RuntimeError (must set SGLANG_PLATFORM) ``` ### Plugin Loading Flow `load_plugins()` discovers and executes general plugins, then applies all registered hooks. It is called at four points:
Call Site Process Timing
cli/serve.py serve() Main Before prepare\_server\_args()
launch\_server.py **main** Main Before prepare\_server\_args()
engine.py \_launch\_subprocesses() Main Before server\_args.check\_server\_args()
scheduler.py run\_scheduler\_process() Subprocess Before Scheduler() construction
> **Note**: `load_plugins()` is idempotent (guarded by `_plugins_loaded` flag). In spawn'd subprocesses the flag resets, so plugins are correctly re-loaded. ``` load_plugins() ├── _get_excluded_dists() → compute dists to skip (via SGLANG_PLATFORM) ├── load_plugins_by_group("sglang.srt.plugins", → discover entry_points, filter by SGLANG_PLUGINS │ excluded_dists=...) skip plugins from unselected platform packages ├── for each plugin: → set _current_plugin_source context var │ func() side effects (register hooks with source tracking) └── HookRegistry.apply_hooks() → monkey-patch targets ``` *** ## Plugin Type 1: Hardware Platform Plugin ### Description A hardware platform plugin registers an `SRTPlatform` subclass that tells SGLang how to interact with a specific hardware backend. ### Quick Start **1. Create a minimal package:** ``` my_platform_plugin/ ├── pyproject.toml └── my_platform_plugin/ ├── __init__.py # activate() function ├── device.py # MyDeviceMixin └── platform.py # MySRTPlatform ``` **2. `pyproject.toml`:** ```toml theme={null} [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "my-platform-plugin" version = "0.1.0" [project.entry-points."sglang.srt.platforms"] my_device = "my_platform_plugin:activate" ``` **3. `__init__.py`** — activation function: ```python theme={null} def activate(): """Return fully-qualified class name to activate, or None to skip.""" if _my_device_is_available(): return "my_platform_plugin.platform.MySRTPlatform" return None ``` **4. `device.py`** — device mixin: ```python theme={null} from sglang.srt.platforms.device_mixin import DeviceMixin, PlatformEnum class MyDeviceMixin(DeviceMixin): _enum = PlatformEnum.OOT device_name = "my_device" device_type = "my_device" # torch device type def set_device(self, device) -> None: ... def get_device_name(self, device_id=0) -> str: ... def get_device_total_memory(self, device_id=0) -> int: ... def get_current_memory_usage(self, device=None) -> float: ... def get_device_capability(self, device_id=0): ... def get_torch_distributed_backend_str(self) -> str: ... ``` **5. `platform.py`** — SRT platform: ```python theme={null} from sglang.srt.platforms.interface import SRTPlatform from my_platform_plugin.device import MyDeviceMixin class MySRTPlatform(SRTPlatform, MyDeviceMixin): def get_default_attention_backend(self) -> str: ... def support_cuda_graph(self) -> bool: ... # ... override other methods as needed ``` **6. Install and verify:** ```bash theme={null} pip install -e my_platform_plugin/ python -c "from sglang.srt.platforms import current_platform; print(current_platform)" ``` ### Platform Interface Reference #### Identity Queries (from DeviceMixin)
Method Default Description
is\_cuda() Based on \_enum Whether this is an NVIDIA CUDA platform
is\_rocm() Based on \_enum Whether this is an AMD ROCm platform
is\_npu() Based on \_enum Whether this is a Huawei NPU platform
is\_cpu() Based on \_enum Whether this is a CPU-only platform
is\_xpu() Based on \_enum Whether this is an Intel XPU platform
is\_musa() Based on \_enum Whether this is a Moore Threads MUSA platform
is\_cuda\_alike() CUDA+ROCM+MUSA True if the hardware supports CUDA-like APIs
is\_out\_of\_tree() True for OOT Automatically detected based on \_enum = PlatformEnum.OOT
#### Device Operations (from DeviceMixin) > Methods annotated **\[Active]** are called by SGLang core through `current_platform` — OOT implementations take effect immediately. > Methods annotated **\[Planned]** are reserved interfaces — SGLang core still uses hardcoded calls (e.g. `torch.cuda.empty_cache()`). OOT implementations will NOT take effect until the core is migrated in a future PR.
Method Default Status Description
get\_device(local\_rank) raise NotImplementedError Planned Return torch.device for a given local rank
set\_device(device) raise NotImplementedError Planned Set the current device
get\_device\_name(device\_id) raise NotImplementedError Planned Get human-readable device name
get\_device\_uuid(device\_id) raise NotImplementedError Planned Get unique device identifier
get\_device\_capability(device\_id) raise NotImplementedError Planned Get DeviceCapability(major, minor). None if N/A
empty\_cache() pass Planned Release cached device memory
synchronize() pass Planned Synchronize device operations
get\_device\_total\_memory(device\_id) raise NotImplementedError Active Get total device memory in bytes
get\_available\_memory(device\_id) raise NotImplementedError Planned Return (free\_bytes, total\_bytes)
get\_current\_memory\_usage(device) raise NotImplementedError Active Get current peak memory usage in bytes
is\_pin\_memory\_available(device=None) False Active Whether pinned host memory is available for a target device
get\_torch\_distributed\_backend\_str() raise NotImplementedError Planned Distributed backend string (e.g. "nccl", "hccl")
get\_communicator\_class() None Planned Platform-specific communicator class
inference\_mode() torch.inference\_mode(True) Planned Return inference mode context manager
seed\_everything(seed) Set random/np/torch seeds Planned Set random seeds for reproducibility
verify\_quantization(quant) pass Planned Validate quantization method support
get\_cpu\_architecture() Auto-detect x86/arm Planned Detect CPU architecture (CpuArchEnum)
#### Types (from DeviceMixin)
Type Description
PlatformEnum Enumeration of platform types: CUDA, ROCM, CPU, XPU, MUSA, NPU, TPU, MPS, OOT, UNSPECIFIED
CpuArchEnum CPU architecture: X86, ARM, UNSPECIFIED
DeviceCapability NamedTuple(major, minor) with comparison support. Methods: as\_version\_str(), to\_int()
#### Capability Flags (from SRTPlatform)
Method Default Description
support\_cuda\_graph() False Whether device graph capture is supported (plain CUDA graph)
support\_piecewise\_cuda\_graph() False Whether piecewise CUDA graph (torch.compile backend) is supported
supports\_fp8() False Whether FP8 quantization is supported
#### Subsystem Factory Methods (from SRTPlatform)
Method Default Description
get\_default\_attention\_backend() raise NotImplementedError Default attention backend name
get\_graph\_runner\_cls() raise NotImplementedError Graph Runner class
get\_mha\_kv\_pool\_cls() raise NotImplementedError MHA KV cache pool class
get\_mla\_kv\_pool\_cls() raise NotImplementedError MLA KV cache pool class
get\_dsa\_kv\_pool\_cls() raise NotImplementedError DSA KV cache pool class (DeepSeek V3.2)
get\_paged\_allocator\_cls() raise NotImplementedError Paged allocator class
get\_quantization\_config(quantization) raise NotImplementedError Return hardware-specific quantization config for the specific quantization scheme, raise an error if not supported or return None to use the default config.
get\_piecewise\_backend\_cls() raise NotImplementedError Piecewise compilation backend class
get\_compile\_backend(mode) "inductor" Compilation backend string
get\_dispatch\_key\_name() "native" BaseFusedOp (fused-op) dispatch key name
#### Lifecycle Hooks (from SRTPlatform)
Method Invocation Timing Purpose
apply\_server\_args\_defaults(server\_args) After ServerArgs parsing, in **post\_init** Set platform-specific defaults
init\_backend() In each worker, before model construction One-time backend initialization
### Environment Variables
Variable Description
SGLANG\_PLATFORM Select the platform plugin by entry\_point name (e.g. kunlun, demo\_cuda). When set, only the named plugin's activate() is called (front-loading filter) — other plugins are not touched. Additionally, general plugins (sglang.srt.plugins) from unselected platform packages are automatically skipped to avoid importing their dependencies. Required when multiple plugins would activate. Errors if the name is not found or if the plugin's hardware is unavailable.
SGLANG\_PLUGINS Comma-separated whitelist of general plugin names to load (group: sglang.srt.plugins). If unset, all discovered general plugins are loaded.
*** ## Plugin Type 2: General Plugin ### Description General function plugins inject behavior into sglang **without requiring a custom platform**. Use cases include: * **Observability**: Add logging, metrics, and tracing to any function * **Behavior modification**: Modify function arguments or return values * **Performance profiling**: Add timing to critical functions * **A/B testing**: Replace implementations at runtime ### Quick Start **1. Create a minimal package:** ``` my_general_plugin/ ├── pyproject.toml └── my_general_plugin/ └── __init__.py # register() function ``` **2. `pyproject.toml`:** ```toml theme={null} [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "my-general-plugin" version = "0.1.0" [project.entry-points."sglang.srt.plugins"] my_plugin = "my_general_plugin:register" ``` **3. `__init__.py`** — register hooks: ```python theme={null} from sglang.srt.plugins.hook_registry import HookRegistry, HookType def register(): """Entry point called by load_plugins().""" HookRegistry.register( "sglang.srt.managers.scheduler.Scheduler.__init__", my_hook, HookType.AROUND, ) def my_hook(original_fn, self, *args, **kwargs): result = original_fn(self, *args, **kwargs) print(f"Scheduler initialized! gpu_id={self.gpu_id}") return result ``` **4. Install and run:** ```bash theme={null} pip install -e my_general_plugin/ sglang serve --model-path [options] # Look for "Scheduler initialized!" in logs ``` ### Hook Types `HookRegistry` supports four hook types:
Hook Type Signature Description
BEFORE fn(\*args, \*\*kwargs) -> (args, kwargs) | None Runs before the original. Return None to keep args unchanged, or (args, kwargs) to modify.
AFTER fn(result, \*args, \*\*kwargs) -> new\_result | None Runs after the original. Return None to keep result, or a new value to replace.
AROUND fn(original\_fn, \*args, \*\*kwargs) -> result Wraps the original. You must call original\_fn yourself. Full control over execution.
REPLACE fn(\*args, \*\*kwargs) -> result or class Replace the original function or class entirely. For class targets, pass a replacement class directly — it is substituted via setattr preserving isinstance()/issubclass() semantics.
> **Note**: Only `REPLACE` accepts a class as the hook. Passing a class to `BEFORE`/`AFTER`/`AROUND` raises `TypeError` at registration time. ### Registration API Hooks can be registered using the **imperative API** or the **decorator API**: ```python theme={null} # --- Imperative API --- from sglang.srt.plugins.hook_registry import HookRegistry, HookType def my_timer(original_fn, *args, **kwargs): start = time.perf_counter() result = original_fn(*args, **kwargs) print(f"Elapsed: {time.perf_counter() - start:.3f}s") return result HookRegistry.register( "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run", my_timer, HookType.AROUND, ) # --- Decorator API --- from sglang.srt.plugins.hook_registry import plugin_hook, HookType @plugin_hook( "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run", type=HookType.AROUND, ) def my_timer(original_fn, *args, **kwargs): start = time.perf_counter() result = original_fn(*args, **kwargs) print(f"Elapsed: {time.perf_counter() - start:.3f}s") return result # --- Class replacement (REPLACE) --- from sglang.srt.plugins.hook_registry import plugin_hook, HookType from sglang.srt.managers.scheduler import Scheduler @plugin_hook( "sglang.srt.managers.scheduler.Scheduler", type=HookType.REPLACE, ) class MyScheduler(Scheduler): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) print("Enhanced scheduler initialized!") ``` ### Hook Target Resolution Target paths use fully-qualified dotted notation. Both formats are supported: * **Dotted**: `sglang.srt.managers.scheduler.Scheduler.__init__` * **Entry-points style**: `sglang.srt.managers.scheduler:Scheduler.__init__` (colon treated as dot) ### Common Hook Targets
Target Description
sglang.srt.server\_args.ServerArgs.add\_cli\_args Add custom CLI arguments
sglang.srt.server\_args.ServerArgs.**post\_init** Modify ServerArgs after parsing
sglang.srt.server\_args.ServerArgs.check\_server\_args Add/relax validation
sglang.srt.managers.scheduler.Scheduler.**init** Custom scheduler state
sglang.srt.managers.scheduler.Scheduler.get\_next\_batch\_to\_run Custom scheduling policy
sglang.srt.managers.scheduler.Scheduler.run\_batch Profiling / inspection
sglang.srt.managers.scheduler.Scheduler.process\_batch\_result Custom metrics
sglang.srt.managers.tp\_worker.TpModelWorker.**init** Custom worker state
sglang.srt.managers.tp\_worker.TpModelWorker.forward\_batch\_generation Forward pass wrapping
*** ## File Reference
File Description
sglang/srt/platforms/device\_mixin.py PlatformEnum + DeviceMixin base class
sglang/srt/platforms/interface.py SRTPlatform base class (extends DeviceMixin)
sglang/srt/platforms/**init**.py current\_platform lazy singleton + discovery logic
sglang/srt/plugins/**init**.py load\_plugins() + load\_plugins\_by\_group()
sglang/srt/plugins/hook\_registry.py HookRegistry, HookType, plugin\_hook decorator
# TPU Source: https://docs.sglang.io/docs/hardware-platforms/tpu SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware. SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware. For TPU-specific issues or feature requests, please visit the [sglang-jax GitHub issues page](https://github.com/sgl-project/sglang-jax/issues). **NOTE:** SGLang TPU support is implemented via the SGLang-JAX backend, a dedicated JAX-based inference engine maintained as a separate repository at [https://github.com/sgl-project/sglang-jax](https://github.com/sgl-project/sglang-jax). ## System Requirements ### Supported TPU Hardware
TPU Type HBM Memory Availability
TPU v6e 32 GB Google Cloud
TPU v7 96 GB per core Google Cloud
### Software Requirements * **Python:** 3.12 or higher * **JAX:** Latest version with TPU support * **Environment:** Google Cloud TPU VM or compatible TPU runtime * **Optional:** SkyPilot for simplified cloud deployment ## Feature Support Matrix SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:
Feature Support Status Description
High-Throughput Continuous Batching ✅ Dynamic request batching for maximum TPU utilization
Radix Tree KV Cache ✅ Memory-efficient prefix sharing between requests
FlashAttention Backend ✅ TPU-optimized attention kernel for long sequences
Tensor Parallelism ✅ Distribute models across multiple TPU cores
Paged Attention ✅ Flexible KV cache management with paging
Speculative Decoding (EAGLE/EAGLE3) ✅ 20-40% throughput improvement for compatible models
Chunked Prefill ✅ Mixed prefill-decode batching
OpenAI-Compatible API ✅ Drop-in replacement for OpenAI API
Data Parallel Attention 🚧 In development - Attention computation with data parallelism
Quantization 🚧 In development - Model quantization for reduced memory usage
Multi-LoRA 🚧 In development - Serve multiple LoRA adapters simultaneously
### Attention Backend Comparison
**Backend** **Paged Attention** **Spec Decoding** **MLA** **Sliding Window**
FlashAttention (fa) ✅ ✅ ❌ ✅
Native ❌ ❌ ❌ ❌
**NOTE:** FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance. ## Optimized Model List The following models have been tested and optimized for TPU deployment:
Model Family Performance Status
Qwen 3 ⭐ Recommended for production
Qwen 3 MoE ⭐ Best performance
Qwen 2 Needs improvement
Qwen 2 MoE Needs improvement
Qwen 1.5 Needs improvement
Llama/LLaMA Needs improvement
Grok-2 Needs improvement
Gemma 2 Verified on TPU
Bailing MoE Needs improvement
## Installation ### Method 1: Using PyPI (Recommended) ```bash Command theme={null} pip install sglang-jax ``` ### Method 2: From Source ```bash Command theme={null} git clone https://github.com/sgl-project/sglang-jax cd sglang-jax uv venv --python 3.12 && source .venv/bin/activate uv pip install -e "python[all]" ``` ### Method 3: Using Docker **NOTE:** Docker support for TPU is currently under development. Please use PyPI or source installation methods. ### Method 4: Cloud TPU with SkyPilot [SkyPilot](https://github.com/skypilot-org/skypilot) provides simplified deployment on Google Cloud TPU: 1. Install SkyPilot and configure GCP access (see [SkyPilot documentation](https://skypilot.readthedocs.io/)) 2. Create a SkyPilot configuration file: SkyPilot YAML: sglang-jax.sky.yaml}> ```yaml Config theme={null} # sglang-jax.sky.yaml resources: accelerators: tpu-v6e-4 accelerator_args: tpu_vm: True runtime_version: v2-alpha-tpuv6e run: | git clone https://github.com/sgl-project/sglang-jax.git cd sglang-jax uv venv --python 3.12 source .venv/bin/activate uv pip install -e "python[all]" ``` 3. Launch your TPU cluster: ```bash Command theme={null} # Standard deployment sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp # With spot instances for cost savings sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot ``` ## Launch of the Serving Engine ### Basic Example: Qwen-7B ```bash Command theme={null} JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen-7B-Chat \ --trust-remote-code \ --dist-init-addr=0.0.0.0:10011 \ --nnodes=1 \ --tp-size=4 \ --device=tpu \ --random-seed=3 \ --node-rank=0 \ --mem-fraction-static=0.8 \ --max-prefill-tokens=8192 \ --download-dir=/tmp \ --dtype=bfloat16 \ --skip-server-warmup \ --host 0.0.0.0 \ --port 30000 ``` **Key Parameters Explained:** 1. `JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache` - Enables JIT compilation caching to accelerate server startup on subsequent runs 2. `--tp-size=4` - Tensor parallelism size; match this to your TPU core count (typically 1, 4, or 8) 3. `--device=tpu` - Specifies TPU device (this is the default for sglang-jax) 4. `--dtype=bfloat16` - Uses bfloat16 precision, which TPUs are optimized for 5. `--mem-fraction-static=0.8` - Allocates 80% of TPU HBM for static memory (adjustable from 0.2 to 0.9) 6. `--max-prefill-tokens=8192` - Maximum number of tokens processed in the prefill phase ### High-Performance Configuration: Qwen3-8B For production workloads with optimal throughput: ```bash Command theme={null} python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen3-8B \ --trust-remote-code \ --tp-size=4 \ --device=tpu \ --mem-fraction-static=0.8 \ --chunked-prefill-size=2048 \ --dtype=bfloat16 \ --max-running-requests=256 \ --page-size=128 \ --attention-backend=fa ``` ### Advanced: Speculative Decoding (EAGLE3) Speculative decoding can improve throughput by 20-40% for compatible models: ```bash Command theme={null} python3 -u -m sgl_jax.launch_server \ --model-path Qwen/Qwen3-32B \ --trust-remote-code \ --device=tpu \ --tp-size=4 \ --mem-fraction-static=0.8 \ --max-prefill-tokens=4096 \ --attention-backend=fa \ --dtype=bfloat16 \ --port=30000 \ --host=0.0.0.0 \ --disable-overlap-schedule \ --speculative-algorithm=EAGLE3 \ --speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \ --page-size=64 \ --speculative-eagle-topk=1 \ --speculative-num-steps=3 \ --speculative-num-draft-tokens=4 ``` **NOTE:** Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration guidance. ### Multi-Node Distributed Serving For large models requiring multiple TPU VMs: ```bash Command theme={null} # Node 0 (coordinator) python3 -m sgl_jax.launch_server \ --model-path MODEL_PATH \ --dist-init-addr=NODE0_IP:10011 \ --nnodes=2 \ --node-rank=0 \ --tp-size=8 \ [other parameters...] # Node 1 (worker) python3 -m sgl_jax.launch_server \ --model-path MODEL_PATH \ --dist-init-addr=NODE0_IP:10011 \ --nnodes=2 \ --node-rank=1 \ --tp-size=8 \ [other parameters...] ``` ## Benchmarking with Requests ### Throughput Testing Basic throughput benchmark: ```bash Command theme={null} python3 -m sgl_jax.bench_serving \ --backend sgl-jax \ --dataset-name random \ --num-prompts=100 \ --random-input=512 \ --random-output=128 \ --max-concurrency=8 \ --random-range-ratio=1 \ --warmup-requests=0 ``` ### Latency Testing Measure single-batch latency: ```bash Command theme={null} python3 -m sgl_jax.bench_one_batch_server \ --base-url http://127.0.0.1:30000 \ --model-path Qwen/Qwen-7B-Chat \ --batch-size=32 \ --input-len=256 \ --output-len=32 ``` ### Comprehensive Benchmark Script For systematic performance evaluation across different configurations: ```bash Command theme={null} #!/bin/bash set -e backend=${1:-sgl-jax} num_prompts_per_concurrency=3 input_seq_lens=(1024 4096 8192) output_seq_lens=(1 1024) max_concurrencies=(8 16 32 64 128 256) for input_seq_len in "${input_seq_lens[@]}"; do for output_seq_len in "${output_seq_lens[@]}"; do echo "=======================================" echo "Testing ISL/OSL: $input_seq_len/$output_seq_len" echo "=======================================" for max_concurrency in "${max_concurrencies[@]}"; do num_prompts=$((num_prompts_per_concurrency * max_concurrency)) python3 -m sgl_jax.bench_serving \ --backend ${backend} \ --dataset-name random \ --num-prompts ${num_prompts} \ --random-input ${input_seq_len} \ --random-output ${output_seq_len} \ --max-concurrency ${max_concurrency} \ --random-range-ratio 1 \ --disable-ignore-eos \ --warmup-requests 0 done done done ``` For detailed help on all benchmark parameters: ```bash Command theme={null} python3 -m sgl_jax.bench_serving --help ``` See the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for advanced benchmarking techniques and profiling with JAX Profiler. ## Performance Optimization ### Memory Optimization **Reduce memory usage:** * Lower `--mem-fraction-static` (from 0.8 → 0.5 → 0.3) * Decrease `--max-prefill-tokens` (from 16384 → 8192 → 4096) * Reduce `--max-running-requests` **Handle OOM errors:** * Start with conservative memory settings (`--mem-fraction-static=0.5`) * Gradually increase until you find the optimal balance * Increase `--page-size` for better memory locality (1 → 16 → 64 → 128) ### Throughput Optimization To maximize tokens per second: * Use FlashAttention backend: `--attention-backend=fa` * Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement) * Increase `--max-running-requests` to 256+ * Set `--mem-fraction-static` to 0.8+ (if memory allows) * Use larger page sizes (64-128) * Enable chunked prefill: `--chunked-prefill-size=2048` ### Latency Optimization To minimize time-to-first-token (TTFT) and inter-token latency: * Reduce `--page-size` to 1-4 * Lower `--max-running-requests` (16-32) for smaller batches * Reduce `--chunked-prefill-size` * Use conservative memory settings to avoid GC pauses ### TPU-Specific Optimizations 1. **JIT Compilation Cache:** ```bash Command theme={null} export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache ``` Always set this environment variable to cache compiled kernels and accelerate server startup. 2. **Data Type Optimization:** Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations. 3. **Tensor Parallelism:** Match `--tp-size` to your TPU core configuration (1, 4, or 8) for optimal model distribution. 4. **Attention Backend:** Always use `--attention-backend=fa` (FlashAttention) for production workloads. ## Troubleshooting ### OOM (Out of Memory) Errors If you encounter out-of-memory errors: 1. Reduce `--mem-fraction-static` from 0.8 to 0.5 or lower 2. Decrease `--max-prefill-tokens` from 8192 to 4096 or 2048 3. Lower `--max-running-requests` to reduce concurrent batch size 4. Increase `--page-size` for better memory layout efficiency ### Compilation Long-Time If the server takes too long to start: 1. Ensure `JAX_COMPILATION_CACHE_DIR` is properly set 2. Understand that the first run requires JIT compilation (this is normal) 3. Subsequent runs will be significantly faster with cached compilations 4. Consider using `--skip-server-warmup` to defer compilation until first request ### Low Throughput If you're not achieving expected throughput: 1. Verify `--tp-size` matches your TPU core configuration 2. Check that `--attention-backend=fa` is enabled 3. Increase `--max-running-requests` to enable larger batch formation 4. Consider enabling speculative decoding for compatible models 5. Ensure memory settings allow for sufficient batch sizes ### Connection Issues If clients cannot connect to the server: 1. Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`) 2. Verify firewall rules allow traffic on the specified port (default: 30000) 3. Check that the server process is running: `curl http://localhost:30000/health` ## Advanced Features ### Speculative Decoding SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration and supported model combinations. ### Chunked Prefill Enable mixed prefill-decode batching for better TPU utilization: ```bash Command theme={null} --chunked-prefill-size=2048 --enable-mixed-chunk ``` This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput. ### Custom Attention Backends SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases. See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/attention_backend.md) for implementation details. ### Environment Verification Verify your TPU setup before deploying: ```bash Command theme={null} python -c "from sgl_jax import check_env; check_env.check_env()" ``` This command checks: * Installed package versions * TPU device availability and specifications * System resources and configuration * Compatibility of settings ## Contributing We welcome contributions to improve TPU support in SGLang-JAX! ### Areas for Contribution **Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax/issues/190)** to see planned features and find opportunities to contribute new functionality. Current contribution areas include: * Performance optimizations for specific TPU generations * Support for additional model architectures * Documentation improvements and examples * Bug reports and fixes * Benchmark results and performance analysis ### How to Contribute 1. Visit the [sglang-jax repository](https://github.com/sgl-project/sglang-jax) 2. Read the [Contribution Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/contribution_guide.md) 3. Join the [SGL-JAX Slack community](https://sgl-fru7574.slack.com/archives/C09EBE5HT5X) for discussions 4. Report issues at [sglang-jax/issues](https://github.com/sgl-project/sglang-jax/issues) ### Testing on TPU For contributors who need TPU access for testing: * Refer to the [TPU Resources Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/tpu_resources_guide.md) for information on accessing TPU hardware * Use SkyPilot with spot instances for cost-effective testing * Follow the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for performance validation ## References ### Documentation * [SGLang-JAX Repository](https://github.com/sgl-project/sglang-jax) * [SGLang-JAX Installation Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/get_started/install.md) * [Qwen Models Quick Start](https://github.com/sgl-project/sglang-jax/blob/main/docs/basic_usage/qwen.md) * [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) * [Speculative Decoding](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) ### External Resources * [JAX Documentation](https://jax.readthedocs.io/) * [Google Cloud TPU Documentation](https://cloud.google.com/tpu/docs) * [SkyPilot Documentation](https://skypilot.readthedocs.io/) # XPU Source: https://docs.sglang.io/docs/hardware-platforms/xpu The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html). Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [ Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html). ## Optimized Model List A list of LLMs have been optimized on Intel GPU, and more are on the way:
Model Name BF16
Llama-3.2-3B [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
Llama-3.1-8B [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
Qwen2.5-1.5B [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
**Note:** The model identifiers listed in the table above have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html). ## Installation ### Install From Source Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency. ```bash Command theme={null} # Create and activate a conda environment conda create -n sgl-xpu python=3.12 -y conda activate sgl-xpu # Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues. pip3 install torch==2.13.0+xpu torchvision==0.28.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU pip3 install apache-tvm-ffi # xgrammar requires apache-tvm-ffi # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git cd sglang git checkout # Use dedicated toml file cd python cp pyproject_xpu.toml pyproject.toml # Install SGLang dependent libs, and build SGLang main package pip install --upgrade pip setuptools pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu ``` ### Install Using Docker [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git cd sglang/docker # Build the docker image docker build -t sglang-xpu:latest -f xpu.Dockerfile . # Initiate a docker container docker run \ -it \ --privileged \ --ipc=host \ --network=host \ --user root \ --group-add $(getent group video | cut -d: -f3) \ --device /dev/dri \ -v /dev/dri/by-path:/dev/dri/by-path \ -v /dev/shm:/dev/shm \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -p 30000:30000 \ -e "HF_TOKEN=" \ sglang-xpu:latest /bin/bash ``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash theme={null} sglang serve \ --model-path \ --trust-remote-code \ --disable-overlap-schedule \ --device xpu \ --host 0.0.0.0 \ --tp 2 \ # using multi GPUs --attention-backend intel_xpu \ # using intel optimized XPU attention backend --page-size \ # intel_xpu attention backend supports [32, 64, 128] ``` ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. Run the command in another terminal. ```bash theme={null} python -m sglang.bench_serving \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 1024 \ --num-prompts 1 \ --request-rate inf \ --random-range-ratio 1.0 ``` The detail explanations of the parameters can be looked up by the command: ```bash theme={null} python -m sglang.bench_serving -h ``` Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script. ## XPU Graph \[Experimental] SGLang enables XPU graph capture to reduce per-step kernel-launch overhead. | Phase | Backend | Mechanism | Default | | ------- | -------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------- | | Decode | `full` | One `torch.xpu.XPUGraph` per batch size, captured on startup | **Off** (opt-in) | | Prefill | `tc_piecewise` | `torch.compile` + XPU graph, one graph segment per token-length bucket | **Off** (opt-in) | | Prefill | `breakable` | Segmented `torch.xpu.XPUGraph` capture/replay (no `torch.compile`); eager break points at attention / MoE boundaries | **Off** (opt-in) | ### Enable Decode Graph Decode graph capture is **opt-in** on XPU. Enable it explicitly: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode full ``` ### Enable Prefill Graph Prefill graph capture is **opt-in** on XPU and must be enabled explicitly. Two backends are available: `tc_piecewise` and `breakable`. #### tc\_piecewise Uses `torch.compile` plus an XPU graph, one graph segment per token-length bucket: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise ``` By default the prefill subgraphs are compiled with `eager` mode. Switch to `inductor` for higher-quality generated code at the cost of longer startup: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise \ --cuda-graph-tc-compiler inductor ``` #### breakable Captures the transformer stack as segmented `XPUGraph`s with eager break points at attention / MoE boundaries, without `torch.compile`: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill breakable ``` You can also configure both phases together with a single `--cuda-graph-config` JSON argument: ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-config '{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}' ``` ### Enable torch.compile for Decode `--enable-torch-compile` adds a `torch.compile` pass on top of the decode XPU graph: the model forward is compiled first, and the compiled forward is then captured as an `XPUGraph`. This can reduce per-kernel overhead further but increases startup time. ```bash theme={null} python -m sglang.launch_server --model-path --device xpu \ --enable-torch-compile ``` > **Note:** `--enable-torch-compile` is mutually exclusive with the prefill > `tc_piecewise` graph (the compatibility rules auto-disable it). Use them > separately or lock the prefill backend explicitly via `--cuda-graph-config` > if you need both. ### Disable XPU Graph Both phases are disabled by default. To explicitly disable them anyway: ```bash theme={null} # Disable decode graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled # Disable prefill graph (already off by default; explicit form) python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-prefill=disabled # Disable both phases python -m sglang.launch_server --model-path --device xpu \ --cuda-graph-backend-decode=disabled \ --cuda-graph-backend-prefill=disabled ``` ### Customize Capture Buckets By default, prefill capture sizes are derived from `--chunked-prefill-size`. To specify explicit token-length buckets: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-backend-prefill tc_piecewise \ --cuda-graph-bs-prefill 64 128 256 512 ``` To specify explicit decode graph batch sizes: ```bash theme={null} python -m sglang.launch_server \ --model-path --device xpu \ --cuda-graph-bs-decode 1 2 4 8 ``` ### Server Args | Argument | XPU allowed values | Default | Description | | ------------------------------ | --------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--cuda-graph-backend-decode` | `full`, `disabled` | `disabled` | Backend for the decode phase. Only `full` is supported on XPU. Set to `full` to enable. | | `--cuda-graph-backend-prefill` | `tc_piecewise`, `breakable`, `disabled` | `disabled`\* | Backend for the prefill phase. Set to `tc_piecewise` or `breakable` explicitly to enable. | | `--cuda-graph-tc-compiler` | `eager`, `inductor` | `eager` | Compiler for `tc_piecewise` prefill subgraphs. `inductor` produces more optimized code but has longer startup. | | `--cuda-graph-bs-prefill` | list of ints | auto | Explicit token-length buckets to capture for prefill. | | `--cuda-graph-bs-decode` | list of ints | auto | Explicit batch sizes to capture for decode. | | `--cuda-graph-config` | JSON string | — | One-shot JSON config for both phases, e.g. `'{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}'`. Overrides all per-phase flags. | | `--disable-decode-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-decode=disabled`. | | `--disable-prefill-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-prefill=disabled`. | | `--enable-torch-compile` | — | `False` | Apply `torch.compile` on top of the decode XPU graph for further kernel optimization. | | `--torch-compile-max-bs` | int | `32` | Maximum batch size compiled by `torch.compile` when `--enable-torch-compile` is set. | \* Prefill graph is auto-disabled on XPU unless you lock the backend explicitly via `--cuda-graph-backend-prefill` or `--cuda-graph-config`. ### Limitations | Feature | Status | | ------------------------------------------------ | ------------------- | | Memory saver (`--enable-memory-saver`) | Not yet supported | | Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported | | Speculative decoding | Not yet implemented | ## Prefill-Decode (P/D) Disaggregation on Intel XPU \[Experimental] SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend. **Tested models:** | Model | Notes | | :-------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------: | | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | | [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) | **Prerequisites:** `pip install nixl sglang-router` **Start the prefill server (GPU 0):** ```bash theme={null} ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode prefill --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000 ``` **Start the decode server (GPU 1):** ```bash theme={null} ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \ --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \ --disaggregation-mode decode --disaggregation-transfer-backend nixl \ --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001 ``` **Start the router:** ```bash theme={null} python -m sglang_router.launch_router \ --pd-disaggregation \ --prefill http://127.0.0.1:30000 \ --decode http://127.0.0.1:30001 \ --host 0.0.0.0 --port 8000 ``` **Send a request:** ```bash theme={null} curl http://127.0.0.1:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}' ``` > **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues. # Custom Chat Template Source: https://docs.sglang.io/docs/references/custom_chat_template **NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat\_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)). By default, the server uses the chat template specified in the model tokenizer from Hugging Face. It should just work for most official models such as Llama-2/Llama-3. If needed, you can also override the chat template when launching the server: ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template llama-2 ``` If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file. ## JSON Format You can load the JSON format, which is defined by `conversation.py`. ```json Config theme={null} { "name": "my_model", "system": "<|im_start|>system", "user": "<|im_start|>user", "assistant": "<|im_start|>assistant", "sep_style": "CHATML", "sep": "<|im_end|>", "stop_str": ["<|im_end|>", "<|im_start|>"] } ``` ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template ./my_model_template.json ``` ## Jinja Format You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers. ```bash Command theme={null} python -m sglang.launch_server \ --model-path meta-llama/Llama-2-7b-chat-hf \ --port 30000 \ --chat-template ./my_model_template.jinja ``` # Environment Variables Source: https://docs.sglang.io/docs/references/environment_variables SGLang supports various environment variables that can be used to configure its runtime behavior. This document provides a comprehensive list and aims to stay updated over time. *Note: The canonical prefix for all SGLang environment variables is `SGLANG_`. The legacy `SGL_` prefix is deprecated: any `SGL_*` variable set in the environment is automatically rewritten to its `SGLANG_*` equivalent at import time with a deprecation warning, and the alias will be removed in a future release. A few variables keep an upstream/vendor prefix (e.g. `MOONCAKE_*`, `ASCEND_*`) because that is their canonical name.* ## General Configuration
Environment Variable Description Default Value
`SGLANG_USE_MODELSCOPE` Enable using models from ModelScope `false`
`SGLANG_HOST_IP` Host IP address for the server `0.0.0.0`
`SGLANG_PORT` Port for the server auto-detected
`SGLANG_LOGGING_CONFIG_PATH` Custom logging configuration path Not set
SGLANG\_LOG\_REQUEST\_HEADERS Comma-separated list of additional HTTP headers to log when --log-requests is enabled. Appends to the default x-smg-routing-key. Not set
SGLANG\_HEALTH\_CHECK\_TIMEOUT Timeout for health check in seconds 20
SGLANG\_EPLB\_HEATMAP\_COLLECTION\_INTERVAL The interval of passes to collect the metric of selected count of physical experts on each layer and GPU rank. 0 means disabled. 0
SGLANG\_EPLB\_P2P\_BATCH\_CHUNK\_SIZE Number of expert IDs per batch when submitting P2P ops during EPLB rebalance (CUDA and ROCm). Smaller values prevent NCCL/RCCL GPU-side accumulation hangs but increase overhead; set >= num\_physical\_experts to submit a single batch. Deprecated alias: SGLANG\_EPLB\_ROCM\_P2P\_BATCH\_CHUNK\_SIZE. 32
SGLANG\_FORWARD\_UNKNOWN\_TOOLS Forward unknown tool calls to clients instead of dropping them false (drop unknown tools)
SGLANG\_REQ\_WAITING\_TIMEOUT Timeout (in seconds) for requests waiting in the queue before being scheduled `-1`
SGLANG\_REQ\_RUNNING\_TIMEOUT Timeout (in seconds) for requests running in the decode batch `-1`
SGLANG\_CACHE\_DIR Cache directory for model weights and other data. Also the default root for compiled-kernel caches: Triton, Inductor, FlashInfer, the CUDA driver and DeepGEMM are pointed under it unless their own env vars (`TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `CUDA_CACHE_PATH`, `SGLANG_DG_CACHE_DIR`) are set explicitly \~/.cache/sglang
SGLANG\_PREFETCH\_BLOCK\_SIZE\_MB Block size (in MB) for sequential checkpoint prefetch reads that warm the OS page cache before workers load weights via mmap 16
## Performance Tuning
Environment Variable Description Default Value
`SGLANG_ENABLE_TORCH_INFERENCE_MODE` Control whether to use torch.inference\_mode `false`
`SGLANG_ENABLE_TORCH_COMPILE` Enable torch.compile false
`SGLANG_SET_CPU_AFFINITY` Enable CPU affinity setting (often set to `1` in Docker builds) false
`SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN` Allows the scheduler to overwrite longer context length requests (often set to `1` in Docker builds) false
`SGLANG_IS_FLASHINFER_AVAILABLE` Control FlashInfer availability check `true`
`SGLANG_FLASHINFER_AUTOTUNE_CACHE` Reuse persisted FlashInfer autotune results from `SGLANG_CACHE_DIR` across runs. Set to `0` to force re-autotuning on every startup; the fresh result is written to a `runs/..json` sibling file (the canonical cache is left untouched). `true`
`SGLANG_SKIP_P2P_CHECK` Skip P2P (peer-to-peer) access check `false`
`SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD` Sets the threshold for enabling chunked prefix caching `8192`
`SGLANG_MAX_KV_CHUNK_CAPACITY` Maximum number of tokens in each KV chunk for DeepSeek MHA chunked prefix cache `131072`
`SGLANG_FUSED_MLA_ENABLE_ROPE_FUSION` Enable RoPE fusion in Fused Multi-Layer Attention `1`
`SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP` Disable overlap schedule for consecutive prefill batches `false`
`SGLANG_SCHEDULER_MAX_RECV_PER_POLL` Set the maximum number of requests per poll, with a negative value indicating no limit `-1`
SGLANG\_MAX\_NEW\_TOKENS\_LIMIT Hard server-side limit for each generation request's `max_new_tokens`; requests asking for more are clipped. Disabled when unset or non-positive. Not set
`SGLANG_DATA_PARALLEL_BUDGET_INTERVAL` Interval for DPBudget updates `1`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DEFAULT` Default weight value for scheduler recv skipper counter (used when forward mode doesn't match specific modes). Only active when --scheduler-recv-interval > 1. The counter accumulates weights and triggers request polling when reaching the interval threshold. `1000`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DECODE` Weight increment for decode forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during decode phase. `1`
SGLANG\_SCHEDULER\_RECV\_SKIPPER\_WEIGHT\_TARGET\_VERIFY Weight increment for target verify forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during verification phase. `1`
`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_NONE` Weight increment when forward mode is None in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency when no specific forward mode is active. `1`
`SGLANG_MM_BUFFER_SIZE_MB` Size of preallocated GPU buffer (in MB) for multi-modal feature hashing optimization. When set to a positive value, temporarily moves features to GPU for faster hash computation, then moves them back to CPU to save GPU memory. Larger features benefit more from GPU hashing. Set to `0` to disable. `0`
`SGLANG_MM_PRECOMPUTE_HASH` Enable precomputing of hash values for MultimodalDataItem `false`
`SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH` Enable NCCL for gathering when preparing mlp sync batch under overlap scheduler (without this flag gloo is used for gathering) `false`
`SGLANG_SYMM_MEM_PREALLOC_GB_SIZE` Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set. -1
SGLANG\_SKIP\_SOFTMAX\_PREFILL\_THRESHOLD\_SCALE\_FACTOR Skip-softmax threshold scale factor for TRT-LLM prefill attention in flashinfer. None means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087) None
SGLANG\_SKIP\_SOFTMAX\_DECODE\_THRESHOLD\_SCALE\_FACTOR Skip-softmax threshold scale factor for TRT-LLM decode attention in flashinfer. None means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087) None
SGLANG\_USE\_SGL\_FA3\_KERNEL Use sgl-kernel implementation for FlashAttention v3 true
## DeepGEMM Configuration (Advanced Optimization)
Environment Variable Description Default Value
`SGLANG_ENABLE_JIT_DEEPGEMM` Enable Just-In-Time compilation of DeepGEMM kernels (enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) GPUs when the DeepGEMM package is installed; set to `"0"` to disable) `"true"`
`SGLANG_JIT_DEEPGEMM_PRECOMPILE` Enable precompilation of DeepGEMM kernels `"true"`
`SGLANG_JIT_DEEPGEMM_COMPILE_WORKERS` Number of workers for parallel DeepGEMM kernel compilation `4`
`SGLANG_IN_DEEPGEMM_PRECOMPILE_STAGE` Indicator flag used during the DeepGEMM precompile script `"false"`
`SGLANG_DG_CACHE_DIR` Directory for caching compiled DeepGEMM kernels `{SGLANG_CACHE_DIR}/deep_gemm`
SGLANG\_DG\_USE\_NVRTC Use NVRTC (instead of Triton) for JIT compilation (Experimental) "false"
SGLANG\_USE\_DEEPGEMM\_BMM Use DeepGEMM for Batched Matrix Multiplication (BMM) operations `"false"`
SGLANG\_JIT\_DEEPGEMM\_FAST\_WARMUP Precompile less kernels during warmup, which reduces the warmup time from 30min to less than 3min. Might cause performance degradation during runtime. `"false"`
## DeepEP Configuration
Environment Variable Description Default Value
`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` The maximum number of dispatched tokens on each GPU `"128"`
`SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK` The maximum number of dispatched tokens on each GPU for --moe-a2a-backend=flashinfer `"1024"`
`SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS` Number of SMs used for DeepEP combine when single batch overlap is enabled `"32"`
`SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` Run shared experts on an alternate stream when single batch overlap is enabled on GB200. When not setting this flag, shared experts and down gemm will be overlapped with DeepEP combine together. `"false"`
`SGLANG_ENABLE_QWEN_DEEPEP_SHARED_OVERLAP` Run the Qwen shared expert on an alternate CUDA stream while DeepEP executes the routed experts. `"true"`
SGLANG\_DISABLED\_MODEL\_ARCHS Comma-separated list of model architectures to disable from auto-registration. Not set
SGLANG\_SORT\_WEIGHT\_FILES Controls weight-file ordering for load-time I/O optimization. -1 disables sorting/staggering (original order); 0 sorts files only; a value k greater than 0 sorts and staggers per-rank order with factor k for better multi-rank I/O concurrency. 0
SGLANG\_RETURN\_ORIGINAL\_LOGPROB Return the original (pre-temperature) logprobs instead of the post-sampling values. false
SGLANG\_ENABLE\_COLOCATED\_BATCH\_GEN Enable colocated batch generation. false
SGLANG\_ENABLE\_MOE\_DEFERRED\_FINALIZE Defer the MoE finalize step to overlap it with other work. true
SGLANG\_PATCH\_TOKENIZER Patch the tokenizer to cache all\_special\_tokens/all\_special\_ids (notably for Kimi tiktoken, where ITL can otherwise regress under high batch size). true
SGLANG\_ENABLE\_LOGITS\_PROCESSER\_CHUNK Process logits in chunks to reduce peak memory. false
SGLANG\_LOGITS\_PROCESSER\_CHUNK\_SIZE Chunk size (in tokens) used when logits-processor chunking is enabled. 2048
SGLANG\_FLASHINFER\_USE\_PAGED Use the paged FlashInfer attention path. false
SGLANG\_FLASHINFER\_WORKSPACE\_SIZE FlashInfer workspace size in bytes (default ≈ 384 MiB). 402653184
SGLANG\_EAGER\_INPUT\_NO\_COPY In eager forward, wrap the ForwardBatch's own tensors instead of copying them into the CUDA graph buffer registry (skips a per-iter device-to-device copy). false
SGLANG\_DEEPGEMM\_SANITY\_CHECK Run extra sanity checks on DeepGEMM kernels. false
SGLANG\_DEEPGEMM\_PDL Enable Programmatic Dependent Launch (PDL) for DeepGEMM kernels. true
SGLANG\_PP\_PARALLEL\_DEEPGEMM\_WARMUP Run DeepGEMM warmup in parallel across pipeline-parallel ranks. false
SGLANG\_DISABLE\_STATIC\_WATERFILL Force dynamic Waterfill with runtime EP all-reduce instead of the default static local-batch path. false
SGLANG\_NIXL\_EP\_BF16\_DISPATCH Use BF16 for NIXL-EP dispatch. false
SGLANG\_NIXL\_EP\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatched tokens per GPU for NIXL-EP. 128
## MORI Configuration
Environment Variable Description Default Value
SGLANG\_MORI\_DISPATCH\_DTYPE Override MoRI-EP dispatch quantization type. auto uses auto-detection from weight dtype; bf16/fp8/fp4 forces the specified type for all layers "auto"
SGLANG\_MORI\_FP8\_COMB Use FP8 for combine "false"
MORI\_DISABLE\_AUTO\_XGMI Set to 0 to allow Mori to automatically use XGMI for same-node PD disaggregation when no active RDMA device is available. unset
SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatch tokens per rank for MORI-EP buffer allocation 4096
SGLANG\_MORI\_DISPATCH\_INTER\_KERNEL\_SWITCH\_THRESHOLD Threshold for switching between InterNodeV1 and InterNodeV1LL kernel types. InterNodeV1LL is used if SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK is less than or equal to this threshold; otherwise, InterNodeV1 is used. 256
SGLANG\_MORI\_PREALLOC\_MAX\_RECV\_TOKENS This argument devives SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK which indicates customized amount of tokens preallocated for a rank, valid range from 1 to world\_size\*SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK, by default 0 means maximum. Setting a smaller value will reduce memory footprint but too small value could cause buffer overflow. 0
SGLANG\_MORI\_MOE\_MAX\_INPUT\_TOKENS Truncate the dispatch buffer to this many rows before MoE computation, reducing kernel overhead on padding tokens. The value must be >= the actual number of received tokens (totalRecvTokenNum); setting it too small causes incorrect results. 0 disables truncation (use full buffer). 0
SGLANG\_PPLX\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK Maximum number of dispatched tokens per rank for the PPLX (--moe-a2a-backend pplx) NVSHMEM buffer allocation. Must be ≥ the worst-case per-rank forward tokens (the per-rank prefill chunk chunked\_prefill\_size / dp\_size, or the decode CUDA-graph batch size); the server refuses to start otherwise. 128
SGLANG\_MORI\_QP\_PER\_TRANSFER Number of RDMA Queue Pairs (QPs) used per transfer operation 1
SGLANG\_MORI\_POST\_BATCH\_SIZE Number of RDMA work requests posted in a single batch to each QP -1
SGLANG\_MORI\_NUM\_WORKERS Number of worker threads in the RDMA executor thread pool 1
## DSA Backend Configuration (For DeepSeek V3.2)
Environment Variable Description Default Value
SGLANG\_DSA\_FUSE\_TOPK Fuse the operation of picking topk logits and picking topk indices from page table. SGLANG\_NSA\_FUSE\_TOPK is a deprecated alias. true
SGLANG\_DSA\_TOPK\_FLASHINFER\_DETERMINISTIC Use deterministic FlashInfer topk kernels when --dsa-topk-backend=flashinfer. false
SGLANG\_DSA\_TOPK\_FLASHINFER\_TIE\_BREAK Tie-break mode for FlashInfer DSA topk when --dsa-topk-backend=flashinfer: unset disables explicit tie-breaking, small prefers the smaller candidate index for equal scores, and large prefers the larger candidate index for equal scores. Setting this variable makes FlashInfer use deterministic topk. unset
SGLANG\_DSA\_PREFILL\_DENSE\_ATTN\_KV\_LEN\_THRESHOLD When the maximum kv len in current prefill batch exceeds this value, the sparse mla kernel will be applied, else it falls back to dense MHA implementation. Default to the index topk of model (2048 for DeepSeek V3.2). SGLANG\_NSA\_PREFILL\_DENSE\_ATTN\_KV\_LEN\_THRESHOLD is a deprecated alias. 2048
SGLANG\_DSA\_TOPK\_BROADCAST Experimental. When enabled, broadcast the finalized NSA/DSA indexer top-k result from attention TP rank 0 to the other attention TP ranks. This can mitigate top-k mismatches in TP attention runs at the cost of some speed. false
SGLANG\_MORI\_SEND\_AUX\_RDMA Send CPU-resident AUX data via RDMA instead of ZMQ TCP. false
SGLANG\_MORI\_TRANSFER\_SHARDS Number of sharded synchronous worker threads draining KV transfers; also bounds outstanding transfers (primary RDMA send-queue throttle). 8
SGLANG\_MORI\_WAIT\_POLL\_MS Poll cadence (ms) at which a transfer worker wakes to check the SLA while waiting for completion. 1000
SGLANG\_MORI\_TRANSFER\_TIMEOUT\_MS Per-transfer SLA (ms) before a KV transfer is failed; 0 disables the SLA. 0
SGLANG\_DSA\_HIP\_DISABLE\_PRESHUFFLE Disable weight pre-shuffle on the HIP DSA path. SGLANG\_NSA\_HIP\_DISABLE\_PRESHUFFLE is a deprecated alias. false
SGLANG\_DSA\_MQA\_LOGITS\_FREE\_MEM\_FRACTION Fraction of free memory the MQA-logits step may use on the DSA path. 0.2
## Memory Management
Environment Variable Description Default Value
SGLANG\_DEBUG\_MEMORY\_POOL Enable memory pool debugging `false`
SGLANG\_CLIP\_MAX\_NEW\_TOKENS\_ESTIMATION Clip max new tokens estimation for memory planning 4096
SGLANG\_DETOKENIZER\_MAX\_STATES Maximum states for detokenizer Default value based on system
SGLANG\_ENABLE\_TP\_MEMORY\_INBALANCE\_CHECK Enable checks for memory imbalance across Tensor Parallel ranks true
SGLANG\_MOONCAKE\_CUSTOM\_MEM\_POOL Configure the custom memory pool type for Mooncake. Supports NVLINK, BAREX, INTRA\_NODE\_NVLINK. If set to true, it defaults to NVLINK. None
## Model-Specific Options
Environment Variable Description Default Value
SGLANG\_USE\_AITER Use AITER optimize implementation `false`
SGLANG\_ROCM\_USE\_MULTI\_STREAM Allocate alt CUDA/HIP stream on ROCm/AITER to overlap shared and routed experts in DeepseekV2 MoE. Requires the HIP env GPU\_MAX\_HW\_QUEUES>=5 (default 4, the cap on HSA/ROCr HW queues HIP creates) so the alt stream gets its own queue instead of serializing with the main stream. Best paired with --deepep-mode low\_latency so Mori's AsyncLL kernel offloads dispatch/combine to copy engines and frees CUs. `false`
SGLANG\_MOE\_PADDING Enable MoE padding (sets padding size to 128 if value is 1, often set to 1 in Docker builds) `false`
SGLANG\_USE\_FUSED\_PARALLEL\_QKNORM Use the fused parallel QK RMSNorm kernel for MiniMax-M2.x on CUDA when attention TP size > 1 `false`
SGLANG\_ENABLE\_STRICT\_MEM\_CHECK\_DURING\_BUSY Enable strict memory checks while the scheduler is busy. 0
SGLANG\_ENABLE\_STRICT\_MEM\_CHECK\_DURING\_IDLE Enable strict memory checks while the scheduler is idle. true
SGLANG\_NATIVE\_MOVE\_KV\_CACHE Use the native implementation to move KV cache entries. false
SGLANG\_USE\_BREAKABLE\_CUDA\_GRAPH Use a breakable CUDA graph so it can be interrupted/rebuilt at runtime. false
SGLANG\_MEMORY\_SAVER\_CUDA\_GRAPH Allow CUDA graphs under the release/resume memory saver. false
SGLANG\_GEMMA\_OUT\_OF\_PLACE\_POSITION\_MUTATION Use out-of-place position mutation for Gemma models. false
SGLANG\_MAMBA\_CONV\_DTYPE dtype for the Mamba convolution state. bfloat16
SGLANG\_MAMBA\_SSM\_DTYPE dtype for the Mamba SSM state (defaults to the model dtype when unset). Not set
SGLANG\_EMBEDDINGS\_SPARSE\_HEAD Name of the sparse-embeddings head to expose for embedding models. Not set
SGLANG\_DSV4\_FP4\_EXPERTS Whether DeepSeek V4 experts use FP4. Set to false when using an FP4-to-FP8 converted DeepSeek V4 checkpoint. true
SGLANG\_DSV4\_REASONING\_EFFORT Default reasoning\_effort for the DeepSeek V4 chat encoder when a request does not set it. The preview profile accepts high and max; the official profile accepts low, high, and max. The profile is detected from the bundled encoder. Override it with --json-model-override-args '\{"dsv4\_reasoning\_effort\_profile":"official"}'. ""
SGLANG\_DSV4\_USE\_BF16\_KV\_QUANT\_SOURCE For DeepSeek V4, quantize the SWA FP8 KV cache from BF16-rounded values instead of FP32 registers. This matches trainer-side QAT and the DSA prefill-CP path, at the cost of an extra BF16 KV materialization and separate cache-store kernels. false
## Quantization
Environment Variable Description Default Value
SGLANG\_INT4\_WEIGHT Enable INT4 weight quantization false
SGLANG\_FORCE\_FP8\_MARLIN Force using FP8 MARLIN kernels even if other FP8 kernels are available false
SGLANG\_NVFP4\_CKPT\_FP8\_GEMM\_IN\_ATTN Quantize q\_b\_proj from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint false
SGLANG\_MOE\_NVFP4\_DISPATCH Use nvfp4 for moe dispatch (on flashinfer\_cutlass or flashinfer\_cutedsl moe runner backend) "false"
SGLANG\_FLASHINFER\_NVFP4\_PER\_TOKEN\_ACTIVATION Enable FlashInfer TRT-LLM or CuTe DSL v2 (no A2A or FlashInfer A2A) per-token FP32 activation scaling for serialized modelopt\_fp4 checkpoints; checkpoint activation scales are treated as 1 false
SGLANG\_FLASHINFER\_MOE\_FUSED\_FINALIZE Use FlashInfer's fused atomic CUTLASS and CuTe DSL MoE finalize for best performance. Deterministic inference overrides this to false. true
FLASHINFER\_NVFP4\_4OVER6 Enable FlashInfer NVFP4 4over6 scaling for NVFP4 per-token activation and online NVFP4 MoE weight quantization paths false
FLASHINFER\_NVFP4\_4OVER6\_E4M3\_USE\_256 Use 256 as the E4M3 scale maximum for FlashInfer NVFP4 4over6 scaling; otherwise uses 448 false
SGLANG\_NVFP4\_CKPT\_FP8\_NEXTN\_MOE Quantize moe of nextn layer from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint `false`
SGLANG\_QUANT\_ALLOW\_DOWNCASTING Allow weight dtype downcasting during loading (e.g., fp32 → fp16). By default, SGLang rejects this kind of downcasting when using quantization. `false`
SGLANG\_FP8\_IGNORED\_LAYERS A comma-separated list of layer names to ignore during FP8 quantization. For example: model.layers.0,model.layers.1.,qkv\_proj. ""
SGLANG\_FP4\_IGNORED\_LAYERS A comma-separated list of layer names to keep out of online FP4 conversion for modelopt\_fp4 or nvfp4\_online. For example: model.layers.40,model.layers.41. ""
SGLANG\_HUMMING\_ONLINE\_QUANT\_CONFIG JSON object or JSON file path for Humming online weight quantization. When a layer has no checkpoint quantization config, this config tells Humming how to quantize the loaded fp16/bf16 weight. When the checkpoint already has a Humming config, add "force\_requant": true to requantize it to this schema during loading. Common keys include dtype/weight\_dtype, scale\_dtype, group\_size, scale\_type, ignored\_layers, ignore, and modules\_to\_not\_convert. None
SGLANG\_HUMMING\_INPUT\_QUANT\_CONFIG JSON object or JSON file path for Humming input activation quantization. This controls the activation dtype and scale grouping passed into Humming kernels, independently of the weight schema. For example, quantizes Humming inputs to FP8 E4M3. None
SGLANG\_HUMMING\_USE\_F16\_ACCUM Use FP16 accumulation in Humming compute/tuning config. This is only meaningful for Humming dtype combinations that support FP16 accumulation, such as fp16 or FP8 E4M3 activations with float16 output. Leave it false for the default accumulator behavior. false
SGLANG\_HUMMING\_MOE\_GEMM\_TYPE Select the Humming MoE GEMM path for standard dispatch and DeepEP normal dispatch. indexed uses top-k expert ids directly and is the fallback for unset or unknown values. grouped maps to Humming grouped-contiguous GEMM. DeepEP low-latency dispatch uses grouped-masked GEMM internally and does not use this selector. "" (indexed)
## Distributed Computing
Environment Variable Description Default Value
`SGLANG_BLOCK_NONZERO_RANK_CHILDREN` Control blocking of non-zero rank children processes `1`
`SGLANG_IS_FIRST_RANK_ON_NODE` Indicates if the current process is the first rank on its node `"true"`
`SGLANG_PP_LAYER_PARTITION` Pipeline parallel layer partition specification Not set
`SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS` Set one visible device per process for distributed computing `false`
SGLANG\_RAY\_BUNDLE\_INDICES Comma-separated bundle indices for Ray actor placement (e.g., "0,1,2,3"). Must match world\_size. Enables fine-grained GPU assignment in custom placement groups. Not set
SGLANG\_CPU\_QUANTIZATION Enable CPU-side quantization. false
SGLANG\_USE\_DYNAMIC\_MXFP4\_LINEAR Use dynamic MXFP4 quantization for linear layers. false
USE\_TRITON\_W8A8\_FP8\_KERNEL Use the Triton W8A8 FP8 kernel. false
SGLANG\_USE\_MESSAGE\_QUEUE\_BROADCASTER Use the shared-memory message-queue broadcaster for inter-process tensor broadcast. true
SGLANG\_DISTRIBUTED\_INIT\_METHOD\_OVERRIDE Override the init method used by torch.distributed.init\_process\_group. Set to env:// to use an externally-created TCPStore via MASTER\_ADDR/MASTER\_PORT. Not set
SGLANG\_TCP\_STORE\_PORT Port for the torch.distributed TCPStore. 29600
SGLANG\_SYNC\_TOKEN\_IDS\_ACROSS\_TP Synchronize sampled token ids across tensor-parallel ranks. false
## PD Disaggregation — Staging Buffer (Heterogeneous TP)
Environment Variable Description Default Value
SGLANG\_DISAGG\_STAGING\_BUFFER Enable GPU staging buffer for heterogeneous TP KV transfer. Required when prefill and decode use different TP/attention-TP sizes. Only for non-MLA models (e.g. GQA, MHA). false
SGLANG\_DISAGG\_STAGING\_POOL\_SIZE\_MB Decode-side ring buffer pool total size in MB. Shared buffer receiving RDMA data from all prefill ranks. Larger values support higher concurrency. 4096
SGLANG\_STAGING\_USE\_TORCH Force using PyTorch gather/scatter fallback instead of Triton fused kernels for staging operations. Useful for debugging. false
## Testing & Debugging (Internal/CI) *These variables are primarily used for internal testing, continuous integration, or debugging.*
Environment Variable Description Default Value
SGLANG\_IS\_IN\_CI Indicates if running in CI environment false
SGLANG\_IS\_IN\_CI\_AMD Indicates running in AMD CI environment false
SGLANG\_TEST\_RETRACT Enable retract decode testing `false`
SGLANG\_TEST\_RETRACT\_NO\_PREFILL\_BS When SGLANG\_TEST\_RETRACT is enabled, no prefill is performed if the batch size exceeds SGLANG\_TEST\_RETRACT\_NO\_PREFILL\_BS. 2 \*\* 31
SGLANG\_RECORD\_STEP\_TIME Record step time for profiling `false`
SGLANG\_TEST\_REQUEST\_TIME\_STATS Test request time statistics `false`
SGLANG\_DEBUG\_SYMM\_MEM Enable debug checks that verify tensors passed to NCCL communication ops are allocated in the symmetric memory pool. Logs warnings (rank 0 only) with stack traces for any tensor not in the pool. `false`
SGLANG\_KERNEL\_API\_LOGLEVEL Controls crash-debug kernel API logging. 0 disables logging, 1 logs API names, 3 logs tensor metadata, 5 adds tensor statistics, and 10 also writes pre-call dump snapshots. 0
SGLANG\_KERNEL\_API\_LOGDEST Destination for crash-debug kernel API logs. Use stdout, stderr, or a file path. %i is replaced with the process PID. stdout
SGLANG\_KERNEL\_API\_DUMP\_DIR Output directory for level-10 kernel API input/output dumps. %i is replaced with the process PID. sglang\_kernel\_api\_dumps
SGLANG\_KERNEL\_API\_DUMP\_INCLUDE Comma-separated wildcard patterns for kernel API names to include in level-10 dumps. Not set
SGLANG\_KERNEL\_API\_DUMP\_EXCLUDE Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps. Not set
## Profiling & Benchmarking
Environment Variable Description Default Value
`SGLANG_TORCH_PROFILER_DIR` Directory for PyTorch profiler output `/tmp`
`SGLANG_PROFILE_WITH_STACK` Set `with_stack` option (bool) for PyTorch profiler (capture stack trace) `true`
`SGLANG_PROFILE_RECORD_SHAPES` Set `record_shapes` option (bool) for PyTorch profiler (record shapes) `true`
`SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS` Config BatchSpanProcessor.schedule\_delay\_millis if tracing is enabled `500`
`SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE` Config BatchSpanProcessor.max\_export\_batch\_size if tracing is enabled `64`
`SGLANG_TRACE_ASYNC` Enable async tracing: span creation is offloaded to a dedicated exporter process via ZMQ, reducing OTel overhead on the inference hot path `false`
`SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD` Number of buffered trace operations before an automatic flush to the exporter process `100`
SGLANG\_PROFILE\_V2 Use the v2 profiler implementation. false
SGLANG\_DETECT\_SLOW\_RANK Detect and report ranks that fall behind during collective ops. false
SGLANG\_ENABLE\_RANK\_CONSENSUS\_CHECKER Check for PP/TP-rank divergence. Kill the server when divergence occurs. Helpful for trouble-shooting server hangs issues. False
SGLANG\_FORCE\_SHUTDOWN Force an immediate process-group shutdown on exit. false
SGLANG\_PYSPY\_DUMP\_BEFORE\_CRASH Capture a py-spy stack dump of all processes before crashing. true
SGLANG\_CUDA\_COREDUMP Enable CUDA coredump generation (auto-injects the required CUDA\_\* env vars). false
SGLANG\_CUDA\_COREDUMP\_DIR Directory for CUDA coredumps. If unset, resolves to RUNNER\_TEMP in CI, else /tmp. Not set
SGLANG\_CUDA\_COREDUMP\_BEFORE\_CRASH Trigger a CUDA coredump before crashing. true
SGLANG\_CUDA\_COREDUMP\_BEFORE\_CRASH\_WAIT\_SECS Seconds to wait for the CUDA coredump to finish before exiting. 60.0
## Storage & Caching
Environment Variable Description Default Value
SGLANG\_WAIT\_WEIGHTS\_READY\_TIMEOUT Timeout period for waiting on weights 120
SGLANG\_DISABLE\_OUTLINES\_DISK\_CACHE Disable Outlines disk cache false
SGLANG\_USE\_CUSTOM\_TRITON\_KERNEL\_CACHE Use SGLang's custom Triton kernel cache implementation for lower overheads (automatically enabled on CUDA) false
SGLANG\_HICACHE\_DECODE\_OFFLOAD\_STRIDE Decode-side incremental KV cache offload stride. Rounded down to a multiple of --page-size (min is --page-size). If unset/invalid/\<=0, it falls back to --page-size. Not set (uses --page-size)
SGLANG\_HICACHE\_NIXL\_USE\_DIRECT\_IO Enable O\_DIRECT for any file-based NIXL backend (POSIX, GDS, GDS\_MT, 3FS) when opening cache files (bypasses the OS page cache, reducing memory pressure and improving throughput on NVMe). Can also be disabled via in --hicache-storage-backend-extra-config. Falls back to buffered I/O with a warning when O\_DIRECT is unavailable on the current OS. true
SGLANG\_HUGEPAGE\_SIZE Use huge pages for host KV cache allocations (HiCache / disaggregation offload). Valid values: 2MB (2 MiB pages via MAP\_HUGE\_2MB) or 1GB (1 GiB pages via MAP\_HUGE\_1GB). Requires huge pages to be pre-allocated on the host OS (/proc/sys/vm/nr\_hugepages or /sys/kernel/mm/hugepages). If the allocation fails, the allocator logs a warning and falls back to regular page-size mmap automatically. Not set (uses OS default page size)
SGLANG\_HICACHE\_HF3FS\_CONFIG\_PATH Path to the HiCache HF3FS backend config file. Not set
SGLANG\_HICACHE\_FILE\_BACKEND\_STORAGE\_DIR Storage directory for the HiCache file backend. Not set
SGLANG\_HICACHE\_FILE\_BACKEND\_MAX\_SIZE Max size for HiCache file-backend LRU eviction (accepts SI/IEC suffixes; 0 disables eviction). Not set (eviction off)
SGLANG\_HICACHE\_FILE\_BACKEND\_EVICTION\_RATIO Target fraction to evict down to when the file-backend max size is reached. 0.9
SGLANG\_HICACHE\_FILE\_BACKEND\_MIN\_FREE\_SPACE Minimum free space to keep on the file-backend volume (accepts SI/IEC suffixes). 0
SGLANG\_HICACHE\_NIXL\_BACKEND\_STORAGE\_DIR Storage directory for the HiCache NIXL backend. Not set
## Function Calling / Tool Use
Environment Variable Description Default Value
SGLANG\_TOOL\_STRICT\_LEVEL Controls the strictness level of tool call parsing and validation. \
Level 0: Off - No strict validation \
Level 1: Function strict - Enables structural tag constraints for all tools (even if none have strict=True set) \
Level 2: Parameter strict - Enforces strict parameter validation for all tools, treating them as if they all have strict=True set		 0
SGLANG\_DEFAULT\_THINKING Enable model thinking/reasoning output by default. false
SGLANG\_MAX\_THINK\_TOKENS Cap on thinking tokens. Negative means unlimited; 0 or greater caps the count. -1
## Logging & Observability
Environment Variable Description Default Value
SGLANG\_LOG\_GC Log Python garbage-collection pauses. false
SGLANG\_LOG\_FORWARD\_ITERS Log each forward iteration. false
SGLANG\_LOG\_MS Log per-step timing in milliseconds. false
SGLANG\_LOG\_REQUEST\_EXCEEDED\_MS Log requests whose processing time exceeds this many milliseconds. -1 disables. -1
SGLANG\_LOG\_SCHEDULER\_STATUS\_TARGET Target (e.g. a file path) for periodic scheduler-status logging. ""
SGLANG\_LOG\_SCHEDULER\_STATUS\_INTERVAL Interval (seconds) between scheduler-status log lines. 60.0
## Constrained Decoding (Grammar)
Environment Variable Description Default Value
SGLANG\_GRAMMAR\_POLL\_INTERVAL Poll interval (seconds) for asynchronous grammar compilation. 0.005
SGLANG\_GRAMMAR\_MAX\_POLL\_ITERATIONS Maximum poll iterations before grammar compilation is treated as stuck. 10000
## Scheduler & Batching
Environment Variable Description Default Value
SGLANG\_INIT\_NEW\_TOKEN\_RATIO Initial new-token ratio used for memory planning. 0.7
SGLANG\_MIN\_NEW\_TOKEN\_RATIO\_FACTOR Floor factor for the new-token ratio after decay. 0.14
SGLANG\_NEW\_TOKEN\_RATIO\_DECAY\_STEPS Number of steps over which the new-token ratio decays. 600
SGLANG\_RETRACT\_DECODE\_STEPS Number of decode steps to look ahead when deciding to retract. 20
SGLANG\_EMPTY\_CACHE\_INTERVAL Interval (seconds) at which to empty the device cache; set this if memory accumulates over a long serving period. -1 disables. -1
SGLANG\_FORCE\_STREAM\_INTERVAL For non-streaming requests, flush intermediate output batches to the tokenizer manager every N decoded tokens (lower to 1 for accurate TTFT benchmarking). 50
SGLANG\_DYNAMIC\_CHUNKING\_SMOOTH\_FACTOR Smoothing factor for dynamic prefill chunking. 0.75
SGLANG\_SWA\_EVICTION\_INTERVAL\_MULTIPLIER Multiplier applied to the sliding-window-attention eviction interval. 1.0
SGLANG\_ENABLE\_UNIFIED\_RADIX\_TREE Use the unified radix-tree cache implementation. false
SGLANG\_EXPERIMENTAL\_CPP\_RADIX\_TREE Use the experimental C++ radix-tree implementation. false
SGLANG\_RADIX\_FORCE\_MISS Force radix-cache misses (debugging/benchmarking). false
SGLANG\_SCHEDULER\_SKIP\_ALL\_GATHER Skip the scheduler all-gather step. false
SGLANG\_ENABLE\_WAR\_BARRIER Force-enable the write-after-read barrier for the overlap scheduler even when CUDA is not detected (e.g. AMD/ROCm). On CUDA the barrier is always enabled. false
SGLANG\_PP\_SKIP\_PURE\_CHUNKED\_OUTPUT\_COMM In pipeline parallel, skip output send/recv when a batch is entirely non-final chunked-prefill requests. false
SGLANG\_KILLPG\_ON\_SCHEDULER\_EXCEPTION Kill the whole process group when the scheduler raises an exception. false
SGLANG\_REQUEST\_STATE\_WAIT\_TIMEOUT Tokenizer-manager request-state wait timeout (seconds). 4
## PD Disaggregation (Runtime)
Environment Variable Description Default Value
SGLANG\_DISAGGREGATION\_THREAD\_POOL\_SIZE Thread-pool size for KV transfers. Defaults to a value computed from the CPU count at runtime. Not set (computed at runtime)
SGLANG\_DISAGGREGATION\_QUEUE\_SIZE Disaggregation transfer queue size. 4
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_TIMEOUT Timeout (seconds) for the disaggregation bootstrap handshake. 300
SGLANG\_DISAGGREGATION\_WAITING\_TIMEOUT Timeout (seconds) for a request waiting on KV transfer. 300
SGLANG\_DISAGGREGATION\_HEARTBEAT\_INTERVAL Interval (seconds) between disaggregation heartbeats. 5.0
SGLANG\_DISAGGREGATION\_HEARTBEAT\_MAX\_FAILURE Consecutive heartbeat failures tolerated before a peer is considered dead. 2
SGLANG\_DISAGGREGATION\_ZMQ\_SEND\_TIMEOUT Send timeout (seconds) for decode's ZMQ sockets to prefill peers. Raise if healthy sends exceed the default. 1
SGLANG\_DISAGGREGATION\_BOOTSTRAP\_ENTRY\_CLEANUP\_INTERVAL Interval (seconds) for cleaning up stale bootstrap entries. 120
SGLANG\_DISAGGREGATION\_NIXL\_BACKEND NIXL transport backend for disaggregation. UCX
SGLANG\_DISAGGREGATION\_NIXL\_BACKEND\_PARAMS JSON parameters passed to the NIXL backend.
SGLANG\_DISAGGREGATION\_ALL\_CP\_RANKS\_TRANSFER Have all context-parallel ranks participate in KV transfer. false
SGLANG\_DISAGGREGATION\_FORCE\_QUERY\_PREFILL\_DP\_RANK Force querying the prefill DP rank for routing. false
SGLANG\_DISAGGREGATION\_NUM\_PRE\_ALLOCATE\_REQS Extra slots in req\_to\_token\_pool for decode workers (effective when max\_num\_reqs greater than 32), letting more KV transfers overlap decode. 0
## Mooncake KV Store & Transfer
Environment Variable Description Default Value
SGLANG\_HICACHE\_MOONCAKE\_CONFIG\_PATH Path to the HiCache Mooncake store config file. Not set
SGLANG\_HICACHE\_MOONCAKE\_REUSE\_TE Reuse the Mooncake transfer engine across HiCache operations. true
SGLANG\_MOONCAKE\_SEND\_AUX\_TCP Send Mooncake AUX data over TCP. false
SGLANG\_ENABLE\_FAILED\_SESSION\_PROBE Probe failed Mooncake sessions for recovery. false
SGLANG\_FAILED\_SESSION\_PROBE\_INTERVAL\_S Interval (seconds) between failed-session probes. 30.0
MOONCAKE\_MASTER Address of the Mooncake master. Not set
MOONCAKE\_CLIENT Mooncake client identifier. Not set
MOONCAKE\_LOCAL\_HOSTNAME Local hostname advertised to Mooncake. localhost
MOONCAKE\_TE\_META\_DATA\_SERVER Mooncake transfer-engine metadata server. P2PHANDSHAKE
MOONCAKE\_GLOBAL\_SEGMENT\_SIZE Mooncake global segment size. 4gb
MOONCAKE\_PROTOCOL Mooncake transport protocol. rdma
MOONCAKE\_DEVICE Mooncake RDMA device(s). ""
MOONCAKE\_MASTER\_METRICS\_PORT Port for Mooncake master metrics. 9003
MOONCAKE\_CHECK\_SERVER Check connectivity to the Mooncake server on startup. false
MOONCAKE\_STANDALONE\_STORAGE Run Mooncake in standalone storage mode. false
MOONCAKE\_ENABLE\_SSD\_OFFLOAD Enable SSD offload in Mooncake. false
MOONCAKE\_OFFLOAD\_FILE\_STORAGE\_PATH File storage path for Mooncake SSD offload. Not set
ENABLE\_ASCEND\_TRANSFER\_WITH\_MOONCAKE Enable Ascend NPU transfers via Mooncake. false
ASCEND\_NPU\_PHY\_ID Physical Ascend NPU id used for Mooncake transfers. -1 auto-detects. -1
## Attention & Kernels
Environment Variable Description Default Value
SGLANG\_TRITON\_DECODE\_ATTN\_STATIC\_KV\_SPLITS Use static KV splits in the Triton decode-attention kernel. false
SGLANG\_MUSA\_FA3\_FORCE\_UPDATE\_METADATA Force FA3 metadata updates on MThreads MUSA. false
SGLANG\_SKIP\_SGL\_KERNEL\_VERSION\_CHECK Skip the sgl-kernel version compatibility check. false
## Deterministic Inference
Environment Variable Description Default Value
SGLANG\_ENABLE\_DETERMINISTIC\_INFERENCE Enable deterministic inference (fixed reduction/accumulation order). false
SGLANG\_USE\_1STAGE\_ALLREDUCE Use the 1-stage all-reduce kernel on AMD (deterministic, fixed accumulation order). If unset, it is auto-enabled when deterministic inference is on. false
SGLANG\_FLASHINFER\_PREFILL\_SPLIT\_TILE\_SIZE FlashInfer prefill split-tile size for deterministic attention. 4096
SGLANG\_FLASHINFER\_DECODE\_SPLIT\_TILE\_SIZE FlashInfer decode split-tile size for deterministic attention. 2048
SGLANG\_TRITON\_PREFILL\_TRUNCATION\_ALIGN\_SIZE Triton prefill truncation alignment size for deterministic attention. 4096
SGLANG\_TRITON\_DECODE\_SPLIT\_TILE\_SIZE Triton decode split-tile size for deterministic attention. 256
## Speculative Decoding & Overlap
Environment Variable Description Default Value
SGLANG\_ENABLE\_OVERLAP\_PLAN\_STREAM Plan the next step on a separate stream to overlap with the current step (Overlap Spec V2). false
SGLANG\_SPEC\_SKIP\_ZERO\_STEP\_DRAFT\_EXTEND Skip draft\_extend while adaptive spec is at steps=0; saves a draft forward but the draft KV goes stale. false
SGLANG\_NGRAM\_FORCE\_GREEDY\_VERIFY Force greedy verification for the n-gram speculative path. false
SGLANG\_SANITIZE\_NAN\_LOGITS Sanitize NaN logits before sampling kernels and emit a throttled warning. true
## EPLB (Expert Parallel Load Balancing)
Environment Variable Description Default Value
SGLANG\_EXPERT\_DISTRIBUTION\_RECORDER\_DIR Output directory for the expert-distribution recorder. /tmp
SGLANG\_ENABLE\_EPLB\_BALANCEDNESS\_METRIC Removed. Use --expert-balancedness-report-mode=prometheus to emit the EPLB balancedness metric. false
SGLANG\_LOG\_EXPERT\_LOCATION\_METADATA Log expert-location metadata. false
SGLANG\_EXPERT\_LOCATION\_UPDATER\_LOG\_INPUT Log inputs to the expert-location updater. false
SGLANG\_EXPERT\_LOCATION\_UPDATER\_LOG\_METRICS Log metrics from the expert-location updater. false
## AMD & ROCm
Environment Variable Description Default Value
SGLANG\_USE\_AITER\_AG Use the AITER all-gather implementation. true
SGLANG\_USE\_AITER\_UNIFIED\_ATTN Use the AITER unified attention kernel. false
SGLANG\_USE\_AITER\_FP8\_PER\_TOKEN Use AITER FP8 per-token quantization. false
SGLANG\_USE\_AITER\_MOE\_GU\_ITLV Select the AITER MoE gate/up tile layout: true interleaves, false uses the separated layout. true
SGLANG\_AITER\_FUSE\_RMSNORM\_PAD Fuse the residual-add + RMSNorm + zero-pad triplet before the MoE block via the AITER Triton kernel (TP=1, post-attention layernorm path only). false
SGLANG\_AITER\_KV\_CACHE\_LAYOUT Physical layout for the MHA KV cache on AITER: nhd or vectorized\_5d (SHUFFLE layout enabling pa\_decode\_gluon). nhd
SGLANG\_ROCM\_FUSED\_DECODE\_MLA Use the fused decode MLA kernel on ROCm. false
SGLANG\_ROCM\_DISABLE\_LINEARQUANT Disable linear-layer quantization on ROCm. false
## NPU (Ascend)
Environment Variable Description Default Value
SGLANG\_NPU\_DISABLE\_ACL\_FORMAT\_WEIGHT Disable ACL-format weight conversion on NPU. false
SGLANG\_NPU\_USE\_MULTI\_STREAM Use multiple streams on NPU. false
SGLANG\_NPU\_USE\_MLAPO Use the MLAPO path on NPU. false
SGLANG\_NPU\_FORWARD\_NATIVE\_GELUTANH Use the native gelu-tanh activation forward (for Skywork-Reward-Gemma-2-27B-v0.2). false
SGLANG\_NPU\_FORWARD\_NATIVE\_GEMMA\_RMS\_NORM Use the native Gemma RMSNorm forward (for Skywork-Reward-Gemma-2-27B-v0.2). false
SGLANG\_USE\_AG\_AFTER\_QLORA Delay all-gather until after QLoRA for better DeepSeek V3.2 performance. false
SGLANG\_EXPERIMENTAL\_LORA\_OPTI Master switch for the experimental TRT-LLM LoRA fast path. When off, all fine-grained opt switches read false. false
SGLANG\_ZBAL\_LOCAL\_MEM\_SIZE Local memory size for the ZBAL (zero-buffer accelerate library) path (NPU only). 0
SGLANG\_ZBAL\_BOOTSTRAP\_URL Bootstrap URL for the ZBAL path (NPU only). ""
## Apple Silicon (MLX / MPS) These variables configure the SRT MLX backend. SGLang Diffusion uses PyTorch MPS and does not read them.
Environment Variable Description Default Value
SGLANG\_USE\_MLX Use the MLX backend on Apple Silicon. false
SGLANG\_MLX\_USE\_CUSTOM\_ROPE Use the custom RoPE kernel on MLX. false
SGLANG\_MLX\_FUSE\_SWIGLU Fuse the SwiGLU activation on MLX. false
SGLANG\_MLX\_CLEAR\_CACHE\_STEPS Number of decode steps between mx.clear\_cache() calls. 0 disables cache clearing. 256
## Multimodal (VLM)
Environment Variable Description Default Value
SGLANG\_VLM\_CACHE\_SIZE\_MB Size (MB) of the VLM feature cache. 100
SGLANG\_IMAGE\_MAX\_PIXELS Maximum number of pixels per image before resizing. 12845056
SGLANG\_RESIZE\_RESAMPLE Resampling filter used when resizing images (e.g. bilinear, bicubic). ""
SGLANG\_MM\_SKIP\_COMPUTE\_HASH Skip computing multimodal-item hashes. false
SGLANG\_MM\_AVOID\_RETOKENIZE For pre-tokenized (list\[int]) multimodal prompts, preserve the user's original tokens to avoid retokenization drift. true
SGLANG\_VIT\_ENABLE\_CUDA\_GRAPH Capture the vision encoder (ViT) in a CUDA graph. false
SGLANG\_USE\_CUDA\_IPC\_TRANSPORT Use CUDA IPC transport for multimodal-item tensors. false
SGLANG\_USE\_IPC\_POOL\_HANDLE\_CACHE When CUDA IPC multimodal feature transport is selected, reuse mappings to its existing bounded pool. This does not enable CUDA IPC transport or reserve another pool. true
SGLANG\_MM\_FEATURE\_CACHE\_MB Size (MB) of the multimodal feature cache. 1024
SGLANG\_MM\_ITEM\_MEM\_POOL\_RECYCLE\_INTERVAL\_SEC Interval (seconds) for recycling the multimodal-item memory pool. 0.05
## Encoder / EPD (Multimodal Disaggregation)
Environment Variable Description Default Value
SGLANG\_ENCODER\_MM\_RECEIVER\_MODE Encoder receiver selection used by EPD paths: http or grpc. http
SGLANG\_ENCODER\_GRPC\_TIMEOUT\_SECS gRPC timeout (seconds) for encoder communication. 60
SGLANG\_ENCODER\_RECV\_TIMEOUT Encoder receive timeout (seconds). 180.0
SGLANG\_ENCODER\_SEND\_TIMEOUT Encoder send timeout (seconds). 180.0
SGLANG\_ENCODER\_HTTP\_TIMEOUT Encoder HTTP timeout (seconds). 1800.0
SGLANG\_ENCODER\_REQ\_TIMEOUT Encoder per-request timeout (seconds). 180.0
SGLANG\_ENCODER\_DISPATCH\_MIN\_ITEMS Minimum items before the encoder dispatches a batch. 2
SGLANG\_ENCODER\_MAX\_BATCH\_SIZE Maximum encoder batch size. 8
SGLANG\_ENCODER\_PREPROC\_WORKERS Number of encoder preprocessing workers. 8
SGLANG\_ENCODER\_IMAGE\_PROCESSOR\_USE\_GPU Run the image processor on the GPU. false
SGLANG\_ENCODER\_BOOTSTRAP\_HEALTH\_CHECK\_INTERVAL EncoderBootstrapServer health-check interval (seconds). 0 disables it. 10.0
SGLANG\_ENCODER\_BOOTSTRAP\_HEALTH\_CHECK\_TIMEOUT EncoderBootstrapServer health-check timeout (seconds). 2.0
SGLANG\_EMBEDDING\_POOL\_SIZE\_MB Persistent receiver-side GPU embedding pool size (MB) for Mooncake EPD transport. 0 disables (per-request register/deregister). 4096
SGLANG\_ENCODER\_DP\_WORKER\_MAX\_INFLIGHT Maximum in-flight requests per encoder DP worker. 64
SGLANG\_BACKUP\_PORT\_BASE Base port for elastic-EP backup ports. 10000
## HTTP & gRPC Server
Environment Variable Description Default Value
SGLANG\_TIMEOUT\_KEEP\_ALIVE HTTP keep-alive timeout (seconds). 5
SGLANG\_UVICORN\_WORKER\_HEALTHCHECK\_TIMEOUT Uvicorn multiprocess supervisor per-worker health-check interval (seconds). 10
SGLANG\_ENABLE\_HEALTH\_ENDPOINT\_GENERATION Have the /health endpoint run a generation as part of the check. true
SGLANG\_WARMUP\_TIMEOUT If a warmup forward batch takes longer than this many seconds, the server crashes to avoid hanging. -1 disables; increase (e.g. to 1800) to accommodate kernel JIT precompile. -1
SGLANG\_GRPC\_PORT Port for the native gRPC server. Not set
SGLANG\_GRANIAN\_PARENT\_PID Parent PID for the Granian HTTP/2 worker supervisor. Not set
## NUMA & CPU
Environment Variable Description Default Value
SGLANG\_NUMA\_BIND\_V2 Select the NUMA binding implementation. true uses a pre-launch numactl wrapper; false binds in the worker process. This variable does not enable or disable NUMA binding. true
SGLANG\_AUTO\_NUMA\_BIND Automatically detect each NVIDIA GPU's local NUMA node and bind its worker when --numa-node is not specified. Set to false to leave workers unbound; an explicit --numa-node takes precedence. true
SGLANG\_CRASH\_ON\_NUMA\_BIND\_FAILURE Crash if NUMA binding fails instead of warning. false
## Metrics
Environment Variable Description Default Value
SGLANG\_ENABLE\_METRICS\_DEVICE\_TIMER Enable device-timer-based metrics. false
SGLANG\_ENABLE\_METRICS\_DP\_ATTENTION Enable data-parallel attention metrics. false
## External Models
Environment Variable Description Default Value
SGLANG\_EXTERNAL\_MODEL\_PACKAGE Python package providing external model implementations. ""
SGLANG\_EXTERNAL\_MM\_MODEL\_ARCH External multimodal model architecture name. ""
SGLANG\_EXTERNAL\_MM\_PROCESSOR\_PACKAGE Python package providing the external multimodal processor. ""
## Plugin System
Environment Variable Description Default Value
SGLANG\_PLATFORM Platform plugin name to load. ""
SGLANG\_PLUGINS Comma-separated list of plugins to load. ""
# Troubleshooting and Frequently Asked Questions Source: https://docs.sglang.io/docs/references/faq ## Troubleshooting This page lists common errors and tips for resolving them. ### CUDA Out of Memory If you encounter out-of-memory (OOM) errors, you can adjust the following parameters: * If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts. * If OOM occurs during decoding, try lowering `--max-running-requests`. * You can also decrease `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput. * Another common case for OOM is requesting input logprobs for a long prompt as it requires significant memory. To address this, set `logprob_start_len` in your sampling parameters to include only the necessary parts. If you do need input logprobs for a long prompt, try reducing `--mem-fraction-static`. ### CUDA Error: Illegal Memory Access Encountered This error may result from kernel errors or out-of-memory issues: * If it is a kernel error, resolving it may be challenging. Please file an issue on GitHub. * If it is an out-of-memory issue, it may sometimes be reported as this error instead of "Out of Memory." Refer to the section above for guidance on avoiding OOM issues. ### The server hangs * If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang. * If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case, you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs-decode`, or decrease `--chunked-prefill-size`. * Other bugs, please file an issue on GitHub. ## Frequently Asked Questions ### The results are not deterministic, even with a temperature of 0 You may notice that when you send the same request twice, the results from the engine will be slightly different, even when the temperature is set to 0. From our initial investigation, this indeterminism arises from two factors: dynamic batching and prefix caching. Roughly speaking, dynamic batching accounts for about 95% of the indeterminism, while prefix caching accounts for the remaining portion. The server runs dynamic batching under the hood. Different batch sizes can cause PyTorch/CuBLAS to dispatch to different CUDA kernels, which can lead to slight numerical differences. This difference accumulates across many layers, resulting in nondeterministic output when the batch size changes. Similarly, when prefix caching is enabled, it can also dispatch to different kernels. Even when the computations are mathematically equivalent, small numerical differences from different kernel implementations lead to the final nondeterministic outputs. To achieve more deterministic outputs in the current code, you can add `--disable-radix-cache` and send only one request at a time. The results will be mostly deterministic under this setting. **Update**: Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`. Please find more details in this blog post: [https://lmsys.org/blog/2025-09-22-sglang-deterministic/](https://lmsys.org/blog/2025-09-22-sglang-deterministic/) # Choices Methods in SGLang Source: https://docs.sglang.io/docs/references/frontend/choices_methods This doc describes the choices methods supported by SGLang. The optional `choices_method` arg determines how options supplied to SGLang's `choices` primitive are selected. Only the `RuntimeEndpoint` backend supports the `choices_method` arg. Other backends, such as `OpenAI`, have bespoke selection implementations due to API limitations. ## Methods ### Token Length Normalized Token length normalized is the default SGLang choices method. It selects the option with the highest average logprob across all of its tokens. Usage example (alternatively, simply omit the `choices_method` arg): ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.token_length_normalized, ) ) ``` This can perform poorly if an option contains many tokens, where its later tokens are predicted with high confidence based on its earlier tokens. For instance, even strong models will fail the above example if the specified options are `["Paris", "Antidisestablishmentarianism"]`. ### Greedy Token Selection Greedy token selection simply selects the option with the highest logprob for its initial token. For overlapping options where one option is a subset of a longer option, the logprobs of the shorter option are extended using its average logprob for comparison against the longer option. Usage example: ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.greedy_token_selection, ) ) ``` This can perform poorly if an option misleads the model down a bad path based on an attractive initial token. For instance, greedy selection will result in an incorrect response for this example: ```python Example theme={null} @sgl.function def us_president_example(s): s += sgl.user("Name a US president.") s += sgl.assistant( sgl.gen( "answer", choices=["Donald Duck", "Millard Fillmore"], choices_method=sgl.greedy_token_selection, ) ) ``` ### Unconditional Likelihood Normalized Unconditional likelihood normalized selects the option with the highest average token logprob once normalized by the unconditional token logprobs, as described in [this EleutherAI blogpost](https://blog.eleuther.ai/multiple-choice-normalization/). This method incurs an additional LLM call to obtain the unconditional likelihoods. Usage example: ```python Example theme={null} @sgl.function def example(s): s += sgl.user("What is the capital of France?") s += sgl.assistant( sgl.gen( "answer", choices=["London", "Paris", "Berlin"], choices_method=sgl.unconditional_likelihood_normalized, ) ) ``` # Frontend Language Source: https://docs.sglang.io/docs/references/frontend/frontend_index * [Frontend Tutorial](./frontend_tutorial) * [Choices Methods](./choices_methods) # SGLang Frontend Language Source: https://docs.sglang.io/docs/references/frontend/frontend_tutorial SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way. ## Launch A Server Launch the server in your terminal and wait for it to initialize. ```python Example theme={null} from sglang import assistant_begin, assistant_end from sglang import assistant, function, gen, system, user from sglang import image from sglang import RuntimeEndpoint from sglang.lang.api import set_default_backend from sglang.srt.utils import load_image from sglang.test.doc_patch import launch_server_cmd from sglang.utils import print_highlight, terminate_process, wait_for_server server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) print(f"Server started on http://localhost:{port}") ``` Set the default backend. Note: Besides the local server, you may use also `OpenAI` or other API endpoints. ```python Example theme={null} set_default_backend(RuntimeEndpoint(f"http://localhost:{port}")) ``` ## Basic Usage The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant. ```python Example theme={null} @function def basic_qa(s, question): s += system(f"You are a helpful assistant than can answer questions.") s += user(question) s += assistant(gen("answer", max_tokens=512)) ``` ```python Example theme={null} state = basic_qa("List 3 countries and their capitals.") print_highlight(state["answer"]) ``` ## Multi-turn Dialog SGLang frontend language can also be used to define multi-turn dialogs. ```python Example theme={null} @function def multi_turn_qa(s): s += system(f"You are a helpful assistant than can answer questions.") s += user("Please give me a list of 3 countries and their capitals.") s += assistant(gen("first_answer", max_tokens=512)) s += user("Please give me another list of 3 countries and their capitals.") s += assistant(gen("second_answer", max_tokens=512)) return s state = multi_turn_qa() print_highlight(state["first_answer"]) print_highlight(state["second_answer"]) ``` ## Control flow You may use any Python code within the function to define more complex control flows. ```python Example theme={null} @function def tool_use(s, question): s += assistant( "To answer this question: " + question + ". I need to use a " + gen("tool", choices=["calculator", "search engine"]) + ". " ) if s["tool"] == "calculator": s += assistant("The math expression is: " + gen("expression")) elif s["tool"] == "search engine": s += assistant("The key word to search is: " + gen("word")) state = tool_use("What is 2 * 2?") print_highlight(state["tool"]) print_highlight(state["expression"]) ``` ## Parallelism Use `fork` to launch parallel prompts. Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel. ```python Example theme={null} @function def tip_suggestion(s): s += assistant( "Here are two tips for staying healthy: " "1. Balanced Diet. 2. Regular Exercise.\n\n" ) forks = s.fork(2) for i, f in enumerate(forks): f += assistant( f"Now, expand tip {i+1} into a paragraph:\n" + gen("detailed_tip", max_tokens=256, stop="\n\n") ) s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n") s += assistant("Tip 2:" + forks[1]["detailed_tip"] + "\n") s += assistant( "To summarize the above two tips, I can say:\n" + gen("summary", max_tokens=512) ) state = tip_suggestion() print_highlight(state["summary"]) ``` ## Constrained Decoding Use `regex` to specify a regular expression as a decoding constraint. This is only supported for local models. ```python Example theme={null} @function def regular_expression_gen(s): s += user("What is the IP address of the Google DNS servers?") s += assistant( gen( "answer", temperature=0, regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)", ) ) state = regular_expression_gen() print_highlight(state["answer"]) ``` Use `regex` to define a `JSON` decoding schema. ```python Example theme={null} character_regex = ( r"""\{\n""" + r""" "name": "[\w\d\s]{1,16}",\n""" + r""" "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n""" + r""" "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n""" + r""" "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n""" + r""" "wand": \{\n""" + r""" "wood": "[\w\d\s]{1,16}",\n""" + r""" "core": "[\w\d\s]{1,16}",\n""" + r""" "length": [0-9]{1,2}\.[0-9]{0,2}\n""" + r""" \},\n""" + r""" "alive": "(Alive|Deceased)",\n""" + r""" "patronus": "[\w\d\s]{1,16}",\n""" + r""" "bogart": "[\w\d\s]{1,16}"\n""" + r"""\}""" ) @function def character_gen(s, name): s += user( f"{name} is a character in Harry Potter. Please fill in the following information about this character." ) s += assistant(gen("json_output", max_tokens=256, regex=character_regex)) state = character_gen("Harry Potter") print_highlight(state["json_output"]) ``` ## Batching Use `run_batch` to run a batch of prompts. ```python Example theme={null} @function def text_qa(s, question): s += user(question) s += assistant(gen("answer", stop="\n")) states = text_qa.run_batch( [ {"question": "What is the capital of the United Kingdom?"}, {"question": "What is the capital of France?"}, {"question": "What is the capital of Japan?"}, ], progress_bar=True, ) for i, state in enumerate(states): print_highlight(f"Answer {i+1}: {states[i]['answer']}") ``` ## Streaming Use `stream` to stream the output to the user. ```python Example theme={null} @function def text_qa(s, question): s += user(question) s += assistant(gen("answer", stop="\n")) state = text_qa.run( question="What is the capital of France?", temperature=0.1, stream=True ) for out in state.text_iter(): print(out, end="", flush=True) ``` ## Complex Prompts You may use `{system|user|assistant}_{begin|end}` to define complex prompts. ```python Example theme={null} @function def chat_example(s): s += system("You are a helpful assistant.") # Same as: s += s.system("You are a helpful assistant.") with s.user(): s += "Question: What is the capital of France?" s += assistant_begin() s += "Answer: " + gen("answer", max_tokens=100, stop="\n") s += assistant_end() state = chat_example() print_highlight(state["answer"]) ``` ```python Example theme={null} terminate_process(server_process) ``` ## Multi-modal Generation You may use SGLang frontend language to define multi-modal prompts. See [here](../../supported-models/multimodal_language_models) for supported models. ```python Example theme={null} server_process, port = launch_server_cmd( "python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning" ) wait_for_server(f"http://localhost:{port}", process=server_process) print(f"Server started on http://localhost:{port}") ``` ```python Example theme={null} set_default_backend(RuntimeEndpoint(f"http://localhost:{port}")) ``` Ask a question about an image. ```python Example theme={null} @function def image_qa(s, image_file, question): s += user(image(image_file) + question) s += assistant(gen("answer", max_tokens=256)) image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png" image_bytes, _ = load_image(image_url) state = image_qa(image_bytes, "What is in the image?") print_highlight(state["answer"]) ``` ```python Example theme={null} terminate_process(server_process) ``` # Deploy On Kubernetes Source: https://docs.sglang.io/docs/references/multi_node_deployment/deploy_on_k8s This document is for deploying a RoCE network-based SGLang two-node inference service on a Kubernetes (K8S) cluster. [LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads. A major use case is for multi-host/multi-node distributed inference. SGLang can also be deployed with LWS on Kubernetes for distributed model serving. Please see this guide for more details on deploying SGLang on Kubernetes using LWS. Here we take the deployment of DeepSeek-R1 as an example. ## Prerequisites 1. At least two Kubernetes nodes, each with two H20 systems and eight GPUs, are required. 2. Make sure your K8S cluster has LWS correctly installed. If it hasn't been set up yet, please follow the [installation instructions](https://lws.sigs.k8s.io/docs/installation/). **Note:** For LWS versions ≤0.5.x, you must use the Downward API to obtain `LWS_WORKER_INDEX`, as native support for this feature was introduced in v0.6.0. ## Basic example For the basic example documentation, refer to [Deploy Distributed Inference Service with SGLang and LWS on GPUs](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sglang). However, that document only covers the basic NCCL socket mode. In this section, we’ll make some simple modifications to adapt the setup to the RDMA scenario. ## RDMA RoCE case * Check your env: ```bash Command theme={null} [root@node1 ~]# ibstatus Infiniband device 'mlx5_bond_0' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe64:c79a base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_1' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe6e:c3ec base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_2' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe73:0dd7 base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet Infiniband device 'mlx5_bond_3' port 1 status: default gid: fe80:0000:0000:0000:0225:9dff:fe36:f7ff base lid: 0x0 sm lid: 0x0 state: 4: ACTIVE phys state: 5: LinkUp rate: 200 Gb/sec (2X NDR) link_layer: Ethernet ``` * Prepare the `lws.yaml` file for deploying on k8s. ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: sglang spec: replicas: 1 leaderWorkerTemplate: size: 2 restartPolicy: RecreateGroupOnPodRestart leaderTemplate: metadata: labels: role: leader spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true hostIPC: true containers: - name: sglang-leader image: sglang:latest securityContext: privileged: true env: - name: NCCL_IB_GID_INDEX value: "3" command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --mem-fraction-static - "0.93" - --torch-compile-max-bs - "8" - --max-running-requests - "20" - --tp - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20000 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --host - "0.0.0.0" - --port - "40000" resources: limits: nvidia.com/gpu: "8" ports: - containerPort: 40000 readinessProbe: tcpSocket: port: 40000 initialDelaySeconds: 15 periodSeconds: 10 volumeMounts: - mountPath: /dev/shm name: dshm - name: model mountPath: /work/models - name: ib mountPath: /dev/infiniband volumes: - name: dshm emptyDir: medium: Memory - name: model hostPath: path: '< your models dir >' # modify it according your models dir - name: ib hostPath: path: /dev/infiniband workerTemplate: spec: dnsPolicy: ClusterFirstWithHostNet hostNetwork: true hostIPC: true containers: - name: sglang-worker image: sglang:latest securityContext: privileged: true env: - name: NCCL_IB_GID_INDEX value: "3" command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --mem-fraction-static - "0.93" - --torch-compile-max-bs - "8" - --max-running-requests - "20" - --tp - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20000 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code resources: limits: nvidia.com/gpu: "8" volumeMounts: - mountPath: /dev/shm name: dshm - name: model mountPath: /work/models - name: ib mountPath: /dev/infiniband volumes: - name: dshm emptyDir: medium: Memory - name: ib hostPath: path: /dev/infiniband - name: model hostPath: path: /data1/models/deepseek_v3_moe *** apiVersion: v1 kind: Service metadata: name: sglang-leader spec: selector: leaderworkerset.sigs.k8s.io/name: sglang role: leader ports: - protocol: TCP port: 40000 targetPort: 40000 ``` * Then use `kubectl apply -f lws.yaml` you will get this output. ```text Output theme={null} NAME READY STATUS RESTARTS AGE sglang-0 0/1 Running 0 9s sglang-0-1 1/1 Running 0 9s ``` Wait for the sglang leader (`sglang-0`) status to change to 1/1, which indicates it is `Ready`. You can use the command `kubectl logs -f sglang-0` to view the logs of the leader node. Once successful, you should see output like this: ```text Output theme={null} [2025-02-17 05:27:24 TP1] Capture cuda graph end. Time elapsed: 84.89 s [2025-02-17 05:27:24 TP6] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP0] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP7] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP3] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP2] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP4] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP1] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24 TP5] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840 [2025-02-17 05:27:24] INFO: Started server process [1] [2025-02-17 05:27:24] INFO: Waiting for application startup. [2025-02-17 05:27:24] INFO: Application startup complete. [2025-02-17 05:27:24] INFO: Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit) [2025-02-17 05:27:25] INFO: 127.0.0.1:48908 - "GET /get_model_info HTTP/1.1" 200 OK [2025-02-17 05:27:25 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0 [2025-02-17 05:27:32] INFO: 127.0.0.1:48924 - "POST /generate HTTP/1.1" 200 OK [2025-02-17 05:27:32] The server is fired up and ready to roll! ``` If it doesn’t start up successfully, please follow these steps to check for any remaining issues. Thanks! ### Debug * Set `NCCL_DEBUG=TRACE` to check if it is a NCCL communication problem. This should resolve most NCCL-related issues. ***Notice: If you find that NCCL\_DEBUG=TRACE is not effective in the container environment, but the process is stuck or you encounter hard-to-diagnose issues, try switching to a different container image. Some images may not handle standard error output properly.*** #### RoCE scenario * Please make sure that RDMA devices are available in the cluster environment. * Please make sure that the nodes in the cluster have Mellanox NICs with RoCE. In this example, we use Mellanox ConnectX 5 model NICs, and the proper OFED driver has been installed. If not, please refer to the document [Install OFED Driver](https://docs.nvidia.com/networking/display/mlnxofedv24102180lts/installing+the+driver) to install the driver. * Check your env: ```shell Command theme={null} $ lspci -nn | grep Eth | grep Mellanox 0000:7f:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:7f:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:c7:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0000:c7:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:08:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:08:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:a2:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) 0001:a2:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01) ``` * Check the OFED driver: ```shell Command theme={null} ofed_info -s OFED-internal-23.07-0.5.0: ``` * Show RDMA link status and check IB devices: ```shell Command theme={null} $ rdma link show 8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0 9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2 10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4 11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6 $ ibdev2netdev 8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0 9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2 10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4 11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6 ``` * Test RoCE network speed on the host: ```shell Command theme={null} yum install qperf # for server： execute qperf # for client qperf -t 60 -cm1 rc_rdma_write_bw ``` * Check RDMA accessible in your container: ```shell Command theme={null} # ibv_devices # ibv_devinfo ``` ## Keys to success * In the YAML configuration above, pay attention to the NCCL environment variable. For older versions of NCCL, you should check the NCCL\_IB\_GID\_INDEX environment setting. * NCCL\_SOCKET\_IFNAME is also crucial, but in a containerized environment, this typically isn’t an issue. * In some cases, it’s necessary to configure GLOO\_SOCKET\_IFNAME correctly. * NCCL\_DEBUG is essential for troubleshooting, but I've found that sometimes it doesn't show error logs within containers. This could be related to the Docker image you're using. You may want to try switching images if needed. * Avoid using Docker images based on Ubuntu 18.04, as they tend to have compatibility issues. ## Remaining issues * In Kubernetes, Docker, or Containerd environments, we use hostNetwork to prevent performance degradation. * We utilize privileged mode, which isn’t secure. Additionally, in containerized environments, full GPU isolation cannot be achieved. ## TODO * Integrated with [k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin). # LWS Based PD Deploy Source: https://docs.sglang.io/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy ## 0. Prerequisites 1. k8s >=1.26 2. lws installed on k8s. ## 1. Image Preparation `lmsysorg/sglang:deepep` ## 2. Deployment Manifest Files ***Notice: We will package all deployment files into Helm Chart format in the near future. Interested community members can contact us to contribute*** ### Prefill Prefill manifest file [prefill.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p.yaml) *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: deepseekr10528-prefill-main spec: leaderWorkerTemplate: leaderTemplate: metadata: labels: role: leader spec: containers: - command: - python3 - -m - sglang.launch_server - --port - "30000" - --host - "0.0.0.0" - --model-path - /work/models - --disaggregation-ib-device # should modify according your rdma env - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 - --chunked-prefill-size - "524288" - --max-prefill-tokens - "32768" - --page-size - "64" # - --init-expert-location # - /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek # - --deepep-config # - /home/aiges/tuned/tuned_8sms.json - --enable-dp-lm-head - --enable-dp-attention - --dp-size - "16" - --disable-radix-cache - --moe-a2a-backend - deepep - --disaggregation-mode - prefill - --mem-fraction-static - "0.7" - --context-length - "32768" - --tp - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: # - name: NVSHMEM_HCA_PE_MAPPING # value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2" # - name: NVSHMEM_HCA_LIST # value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: none - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "false" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-leader ports: - containerPort: 30000 protocol: TCP readinessProbe: periodSeconds: 30 tcpSocket: port: 30000 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf - mountPath: /root/.cache name: sgl-cache dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - emptyDir: medium: Memory name: dshm - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /dev/infiniband name: ib - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf - hostPath: # modify according to you deployment env path: /data1/sgl_cache type: DirectoryOrCreate name: sgl-cache restartPolicy: RecreateGroupOnPodRestart size: 2 workerTemplate: metadata: {} spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --disaggregation-ib-device - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 - --chunked-prefill-size - "524288" - --max-prefill-tokens - "32768" - --page-size - "64" #- --init-expert-location #- /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek # - --deepep-config # - /home/aiges/tuned/tuned_8sms.json - --enable-dp-lm-head - --enable-dp-attention - --dp-size - "16" - --disable-radix-cache - --moe-a2a-backend - deepep - --disaggregation-mode - prefill - --mem-fraction-static - "0.7" - --context-length - "32768" - --tp - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGLANG_HACK_DEEPEP_NUM_SMS value: "8" - name: SGLANG_HACK_DEEPEP_NEW_MODE value: "0" # - name: NVSHMEM_HCA_PE_MAPPING # value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2" # - name: NVSHMEM_HCA_LIST # value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "8" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD value: "0" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: none - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "true" - name: NCCL_IB_SL value: "5" - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-worker ports: - containerPort: 30001 protocol: TCP resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - emptyDir: medium: Memory name: dshm - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf - hostPath: path: /data1/sgl_cache type: DirectoryOrCreate name: sgl-cache ``` ### Decode Decode node deployment manifest file [decode.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d.yaml) *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* ```yaml Config theme={null} apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata: name: deepseekr10528-decode-main spec: leaderWorkerTemplate: leaderTemplate: metadata: labels: role: leader spec: containers: - command: - python3 - -m - sglang.launch_server - --port - "30000" - --host - "0.0.0.0" - --model-path - /work/models - --chunked-prefill-size - "262144" - --page-size - "64" - --enable-dp-attention - --enable-dp-lm-head - --dp-size - "16" - --moe-a2a-backend - deepep - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3" - --cuda-graph-max-bs-decode - "64" - --max-running-requests - "2048" - --tp-size - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: CUDA_LAUNCH_BLOCKING value: "0" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_IB_SL value: "5" - name: MC_TE_METRIC value: "true" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "16" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-leader ports: - containerPort: 30000 protocol: TCP readinessProbe: periodSeconds: 30 tcpSocket: port: 30000 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - hostPath: path: /data1/sgl_cache1 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf restartPolicy: RecreateGroupOnPodRestart size: 2 workerTemplate: metadata: {} spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --chunked-prefill-size - "262144" - --page-size - "64" - --enable-dp-attention - --enable-dp-lm-head #- --enable-two-batch-overlap - --dp-size - "16" - --moe-a2a-backend - deepep - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device # should modify according your rdma env - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3" - --cuda-graph-max-bs-decode - "64" - --max-running-requests - "2048" - --tp-size - "16" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: SGLANG_HACK_DEEPEP_NUM_SMS value: "24" - name: SGLANG_HACK_DEEPEP_NEW_MODE value: "0" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_ENABLE_NIC_PE_MAPPING value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_MIN_NCHANNELS value: "4" - name: MC_TE_METRIC value: "true" - name: NCCL_IB_SL value: "5" - name: SGLANG_MOONCAKE_TRANS_THREAD value: "16" - name: SGLANG_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] image: lmsysorg/sglang:deepep name: sglang-worker ports: - containerPort: 30001 resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs name: cf dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists volumes: - hostPath: path: /data1/sgl_cache1 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /dev/infiniband name: ib - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528 name: model - hostPath: # modify according to you deployment env path: /data1/maas_hosted_models/models/fused_moe_triton/configs name: cf networkConfig: subdomainPolicy: Shared replicas: 1 rolloutStrategy: rollingUpdateConfiguration: maxSurge: 0 maxUnavailable: 1 type: RollingUpdate startupPolicy: LeaderCreated ``` Execute separately: ```bash Command theme={null} kubectl apply -f p.yaml kubectl apply -f d.yaml ``` At this point, we have completed the deployment of the 1P1D SGLang engine part. To allow our users to directly experience the model API, we still need a load balancer to handle sequential calls between prefill and decode. Different companies implement LBs differently, and the community will also officially release a new LB component written in Rust in the near future. Currently, we use a static K8S service + minilb approach to implement model API calls. ### Creating Service for Prefill and Decode #### Create prefill k8s service [p-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p-svc.yaml) ```yaml Config theme={null} apiVersion: v1 kind: Service metadata: name: deepseekr10528-prefill-main spec: selector: leaderworkerset.sigs.k8s.io/name: deepseekr10528-prefill-main role: leader ports: - protocol: TCP port: 30000 targetPort: 30000 ``` Execute `kubectl apply -f p-svc.yaml` #### Create decode k8s service [d-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d-svc.yaml) ```yaml Config theme={null} apiVersion: v1 kind: Service metadata: name: deepseekr10528-decode-main spec: selector: leaderworkerset.sigs.k8s.io/name: deepseekr10528-decode-main role: leader ports: - protocol: TCP port: 30000 targetPort: 30000 ``` Execute `kubectl apply -f d-svc.yaml` #### Deploy minilb and lb service [lb.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/lb.yaml) ```yaml Config theme={null} apiVersion: apps/v1 kind: Deployment metadata: name: deepseekr10528-lb-main labels: app: deepseekr10528-lb spec: replicas: 1 selector: matchLabels: app: deepseekr10528-lb template: metadata: labels: app: deepseekr10528-lb spec: nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists - key: node-role operator: Exists containers: - name: sgl-minilb image: lmsysorg/sglang:deepep command: - python - -m - sglang_router.launch_router - --pd-disaggregation - --prefill - http://deepseekr10528-prefill-main:30000 - --decode - http://deepseekr10528-decode-main:30000 - --host - 0.0.0.0 - --port - "8000" ports: - containerPort: 8000 *** apiVersion: v1 kind: Service metadata: name: deepseekr10528-lb-service spec: type: NodePort selector: app: deepseekr10528-lb ports: - protocol: TCP port: 8000 # Service Port（In-Cluster） targetPort: 8000 # Exposed Container nodePort: 30800 ``` Execute `kubectl apply -f lb.yaml` After waiting for all model deployments to succeed, you will get the following output: ```bash Command theme={null} [root@ecs-001]# kubectl get po deepseekr10528-decode-main-0 1/1 Running 0 74m deepseekr10528-decode-main-0-1 1/1 Running 0 74m deepseekr10528-lb-main-9c5dbfc57-6lcbd 1/1 Running 0 22m deepseekr10528-prefill-main-0 1/1 Running 0 74m deepseekr10528-prefill-main-0-1 1/1 Running 0 74m [root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl get svc |grep dee deepseekr10528-decode-main ClusterIP None 97m deepseekr10528-lb-service NodePort 172.16.242.169 8000:30800/TCP 22m deepseekr10528-prefill-main ClusterIP None 97m ``` At this point, select a nodePort:30800 to access: ```bash Command theme={null} [root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -H "Authorization: Bearer None" \ > -d '{ > "rid":"ccccdd", > "model": "r1", > "messages": [ > {"role": "system", "content": "0: You are a helpful AI assistant"}, > {"role": "user", "content": "你是谁？."} > ], > "max_tokens":221 > }' {"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-R1**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}} ``` ## FAQ 1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments. 2. Some preset, optimized configurations for EPLB are not used here. You can adjust them according to [6017](https://github.com/sgl-project/sglang/issues/6017) as needed. # Multi-Node Deployment Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node ## Llama 3.1 405B **Run 405B (fp16) on Two Nodes** ```bash Command theme={null} # replace 172.16.4.52:20000 with your own node ip address and port of the first node python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \ --tp 16 \ --dist-init-addr 172.16.4.52:20000 \ --nnodes 2 \ --node-rank 0 python3 -m sglang.launch_server \ --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \ --tp 16 \ --dist-init-addr 172.16.4.52:20000 \ --nnodes 2 \ --node-rank 1 ``` Note that LLama 405B (fp8) can also be launched on a single node. ```bash Command theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8 ``` ## DeepSeek V3/R1 Please refer to [DeepSeek documents for reference](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-5-multi-node-deployment). ## Multi-Node Inference on SLURM This example showcases how to serve SGLang server across multiple nodes by SLURM. Submit the following job to the SLURM cluster. ```bash Command theme={null} #!/bin/bash -l #SBATCH -o SLURM_Logs/%x_%j_master.out #SBATCH -e SLURM_Logs/%x_%j_master.err #SBATCH -D ./ #SBATCH -J Llama-405B-Online-Inference-TP16-SGL #SBATCH --nodes=2 #SBATCH --ntasks=2 #SBATCH --ntasks-per-node=1 # Ensure 1 task per node #SBATCH --cpus-per-task=18 #SBATCH --mem=224GB #SBATCH --partition="lmsys.org" #SBATCH --gres=gpu:8 #SBATCH --time=12:00:00 echo "[INFO] Activating environment on node $SLURM_PROCID" if ! source ENV_FOLDER/bin/activate; then echo "[ERROR] Failed to activate environment" >&2 exit 1 fi # Define parameters model=MODEL_PATH tp_size=16 echo "[INFO] Running inference" echo "[INFO] Model: $model" echo "[INFO] TP Size: $tp_size" # Set NCCL initialization address using the hostname of the head node HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1) NCCL_INIT_ADDR="${HEAD_NODE}:8000" echo "[INFO] NCCL_INIT_ADDR: $NCCL_INIT_ADDR" # Launch the model server on each node using SLURM srun --ntasks=2 --nodes=2 --output="SLURM_Logs/%x_%j_node$SLURM_NODEID.out" \ --error="SLURM_Logs/%x_%j_node$SLURM_NODEID.err" \ python3 -m sglang.launch_server \ --model-path "$model" \ --grammar-backend "xgrammar" \ --tp "$tp_size" \ --dist-init-addr "$NCCL_INIT_ADDR" \ --nnodes 2 \ --node-rank "$SLURM_NODEID" & # Wait for the NCCL server to be ready on port 30000 while ! nc -z "$HEAD_NODE" 30000; do sleep 1 echo "[INFO] Waiting for $HEAD_NODE:30000 to accept connections" done echo "[INFO] $HEAD_NODE:30000 is ready to accept connections" # Keep the script running until the SLURM job times out wait ``` Then, you can test the server by sending requests following other [documents](../../basic_usage/openai_api_completions). Thanks for [aflah02](https://github.com/aflah02) for providing the example, based on his [blog post](https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang). # Multi-Node Deployment Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node_index * [Multi Node](./multi_node) * [Deploy On K8S](./deploy_on_k8s) * [Lws Pd Deploy](./lws_pd/lws_pd_deploy) * [Deepseekv32 Pd](./rbg_pd/deepseekv32_pd) * [Deploying DeepSeek with PD Disaggregation on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-large-scale-ep/) * [Deploying Kimi K2 with PD Disaggregation on 128 H200 GPUs](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/) # DeepSeekV32-Exp RBG Based PD Deploy Source: https://docs.sglang.io/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd ## 0. Prerequisites 1. k8s >=1.26 2. lws installed on k8s. 3. rbg installed on k8s. For RBG installation, please refer to: [https://github.com/sgl-project/rbg](https://github.com/sgl-project/rbg) ## 1. Image Preparation `lmsysorg/sglang:latest` ### 2. All In One manifest file *Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment* rbg-dsv32.yml ```yaml Config theme={null} apiVersion: workloads.x-k8s.io/v1alpha1 kind: RoleBasedGroup metadata: name: deepseek-rbg-32exp namespace: default spec: roles: - name: prefill replicas: 1 workload: apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet restartPolicy: None leaderWorkerSet: size: 1 patchLeaderTemplate: metadata: labels: role: leader pd_role: prefill spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --port - "30000" - --trust-remote - --host - 0.0.0.0 - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --disable-radix-cache - --chunked-prefill-size - "131072" - --page-size - "64" # - --enable-eplb - --ep-dispatch-algorithm - dynamic - --eplb-algorithm - deepseek - --enable-dp-lm-head - --enable-dp-attention - --dp-size - "8" - --moe-a2a-backend - deepep - --deepep-mode - normal - --disaggregation-mode - prefill - --mem-fraction-static - "0.8" - --max-prefill-tokens - "32768" - --context-length - "32768" - --tp - "8" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" - --max-running-requests - "1024" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] livenessProbe: failureThreshold: 3000 httpGet: path: /health port: 30000 initialDelaySeconds: 300 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 20 httpGet: path: /health port: 30000 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 10 name: sglang ports: - containerPort: 30000 name: sglang-http protocol: TCP patchWorkerTemplate: {} template: metadata: labels: inference-framework: sglang inference-stack.io/monitoring: "enabled" spec: containers: - name: sglang image: lmsysorg/sglang:latest env: - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK value: "1" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT value: "1000000000" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_DISABLE_P2P value: "0" - name: ENABLE_METRICS value: "true" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NVSHMEM_IB_SL value: "5" - name: SGLANG_SET_CPU_AFFINITY value: "true" - name: SGL_ENABLE_JIT_DEEPGEMM value: "1" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_TIMEOUT value: "22" - name: NCCL_IB_GID_INDEX value: "3" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_SOCKET_IFNAME value: bond0 - name: GLOO_SOCKET_IFNAME value: bond0 - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME value: "bond0" - name: MC_TE_METRIC value: "false" resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang name: src dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists volumes: - hostPath: path: /var/run/sys-topology name: topo - hostPath: path: /data1/sgl_cache4 type: DirectoryOrCreate name: sgl-cache - emptyDir: medium: Memory name: dshm - hostPath: path: /data/DeepSeek-V3.2-Exp name: model - hostPath: path: /dev/infiniband name: ib - hostPath: path: /data/src/sglang type: DirectoryOrCreate name: src - name: decode replicas: 1 workload: apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet leaderWorkerSet: size: 1 patchLeaderTemplate: metadata: labels: role: leader pd_role: decode spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --port - "30000" - --trust-remote - --host - 0.0.0.0 - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --chunked-prefill-size - "131072" - --eplb-rebalance-layers-per-chunk - "29" - --page-size - "64" - --enable-dp-attention - --enable-dp-lm-head - --dp-size - "8" - --moe-a2a-backend - deepep - --deepep-mode - low_latency - --disaggregation-mode - decode - --mem-fraction-static - "0.8" - --context-length - "32768" - --max-running-requests - "2048" - --tp-size - "8" # Size of Tensor Parallelism - --cuda-graph-max-bs-decode - "16" - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] livenessProbe: failureThreshold: 30000 httpGet: path: /health port: 30000 initialDelaySeconds: 300 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 10 name: sglang readinessProbe: failureThreshold: 20 httpGet: path: /health port: 30000 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 10 patchWorkerTemplate: spec: containers: - command: - python3 - -m - sglang.launch_server - --model-path - /work/models - --crash-dump-folder - /log - --chunked-prefill-size - "262144" - --eplb-rebalance-layers-per-chunk - "29" - --page-size - "64" - --enable-dp-attention - --enable-dp-lm-head - --dp-size - "32" - --moe-a2a-backend - "deepep" - --deepep-mode - low_latency - --disaggregation-mode - decode - --mem-fraction-static - "0.849" - --context-length - "32768" - --disaggregation-ib-device - mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 - --max-running-requests - "4096" - --cuda-graph-max-bs-decode - "16" - --tp-size - "8" # Size of Tensor Parallelism - --dist-init-addr - $(LWS_LEADER_ADDRESS):20102 - --nnodes - $(LWS_GROUP_SIZE) - --node-rank - $(LWS_WORKER_INDEX) - --trust-remote-code - --ep-num-redundant-experts - "32" - --moe-dense-tp-size - "1" env: - name: LWS_WORKER_INDEX valueFrom: fieldRef: fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index'] name: sglang template: metadata: labels: inference-framework: sglang-unuse inference-stack.io/monitoring: "enabled" spec: containers: - image: lmsysorg/sglang:latest name: sglang resources: limits: nvidia.com/gpu: "8" securityContext: capabilities: add: - IPC_LOCK privileged: true volumeMounts: - mountPath: /root/.cache name: sgl-cache - mountPath: /dev/shm name: dshm - mountPath: /work/models name: model - mountPath: /dev/infiniband name: ib - mountPath: /sgl-workspace/sglang name: src env: - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK value: "1" - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT value: "100000000" - name: NVSHMEM_DISABLE_P2P value: "0" - name: NVSHMEM_IB_TRAFFIC_CLASS value: "16" - name: NVSHMEM_IB_SL value: "5" - name: ENABLE_METRICS value: "true" - name: CUDA_LAUNCH_BLOCKING value: "0" - name: NVSHMEM_IB_GID_INDEX value: "3" - name: NCCL_IB_QPS_PER_CONNECTION value: "8" - name: NCCL_IB_SPLIT_DATA_ON_QPS value: "1" - name: NCCL_NET_PLUGIN value: "none" - name: NCCL_IB_TC value: "136" - name: NCCL_IB_SL value: "5" - name: NCCL_IB_TIMEOUT value: "22" - name: NCCL_IB_GID_INDEX value: "3" - name: NCCL_MIN_NCHANNELS value: "4" - name: NCCL_SOCKET_IFNAME value: bond0 - name: GLOO_SOCKET_IFNAME value: bond0 - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME value: "bond0" - name: NCCL_IB_HCA value: ^=mlx5_0,mlx5_5,mlx5_6 - name: MC_TE_METRIC value: "false" - name: SGL_ENABLE_JIT_DEEPGEMM value: "1" dnsPolicy: ClusterFirstWithHostNet hostIPC: true hostNetwork: true nodeSelector: pd: "yes" tolerations: - key: pd operator: Exists volumes: - hostPath: path: /var/run/sys-topology name: topo - hostPath: path: /data1/sgl_cache4 type: DirectoryOrCreate name: sgl-cache - hostPath: path: /data/src/sglang type: DirectoryOrCreate name: src - emptyDir: medium: Memory name: dshm - hostPath: path: /data/DeepSeek-V3.2-Exp name: model - hostPath: path: /dev/infiniband name: ib - name: router replicas: 1 dependencies: [ "decode", "prefill" ] template: spec: containers: - name: scheduler image: lmsysorg/sglang:latest command: - sh - -c - > python3 -m sglang_router.launch_router --host 0.0.0.0 --port 8080 --pd-disaggregation --policy random --service-discovery --service-discovery-namespace ${NAMESPACE} --service-discovery-port 30000 --prefill-selector pd_role=prefill --decode-selector pd_role=decode --max-payload-size 2147483648 --worker-startup-timeout-secs 1200 env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace *** apiVersion: v1 kind: Service metadata: labels: app: deepseek-rbg-32exp name: deepseek-rbg-32exp namespace: default spec: ports: - name: http port: 8080 protocol: TCP targetPort: 8080 nodePort: 30080 selector: rolebasedgroup.workloads.x-k8s.io/name: deepseek-rbg-32exp rolebasedgroup.workloads.x-k8s.io/role: router type: NodePort ``` ```bash Command theme={null} [root@ecs-001]# kubectl get po -n default deepseek-rbg-32exp-decode-main-0 1/1 Running 0 74m deepseek-rbg-32exp-decode-0-1 1/1 Running 0 74m deepseek-rbg-32exp-router-9c5dbfc57 1/1 Running 0 22m deepseek-rbg-32exp-prefill-0 1/1 Running 0 74m [root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl get svc |grep dee deepseek-rbg-32exp-decode ClusterIP None 97m deepseek-rbg-32exp-router-service NodePort 172.16.242.169 8000:30800/TCP 22m deepseek-rbg-32exp-prefill ClusterIP None 97m ``` At this point, select a nodePort:30800 to access: ```bash Command theme={null} [root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -H "Authorization: Bearer None" \ > -d '{ > "rid":"ccccdd", > "model": "dsv32", > "messages": [ > {"role": "system", "content": "0: You are a helpful AI assistant"}, > {"role": "user", "content": "你是谁？."} > ], > "max_tokens":221 > }' {"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n</think>\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-V32**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}} ``` ## FAQ 1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments. 2. Please ensure that the sglang code in the image has incorporated the changes from [PR #10912](https://github.com/sgl-project/sglang/pull/10912). # Nightly precision regression Source: https://docs.sglang.io/docs/references/nightly_precision_regression # Nightly Precision Regression Testing ## Overview The nightly precision regression framework detects silent numerical regressions in the SGLang serving engine by comparing **per-layer hidden states** between consecutive runs. It runs as a nightly CI job on 8×H200 GPUs and can also be invoked locally for development and debugging. The framework operates on a **rolling-baseline** model: 1. **Baseline creation or comparison:** Launch the server, send a fixed prompt, dump per-layer hidden states to disk. If a previous baseline exists, compare the new tensors against it using the SGLang tensor comparator. If the comparison passes, the new tensors become the updated baseline. 2. On the first run (or when the capture shape changes), the dumped tensors are saved as a new baseline with no comparison. Baselines are stored locally on disk and synced to a **HuggingFace dataset** so they survive across CI runners and can be shared across machines. The HF dataset store is **required** — the test errors if `SGLANG_PRECISION_HF_REPO` is unset. *** ## How It Works ### Step-by-step flow ``` ┌──────────────────────────────────────────────────────────────┐ │ 1. Resolve model config (layer count, capture layers) │ │ ↓ │ │ 2. Compute capture_signature (schema, layers, TP, filter) │ │ ↓ │ │ 3. Fetch baseline from HF dataset (signature-matched) │ │ ↓ │ │ 4. Launch SGLang server with DUMPER enabled │ │ ↓ │ │ 5. POST /dumper/configure (set layer filter + cleanup) │ │ ↓ │ │ 6. POST /v1/chat/completions (fixed prompt, 2 tokens, │ │ ignore_eos=true to force decode path) │ │ ↓ │ │ 7. Kill server; assert decode tensors were captured │ │ ↓ │ │ 8. Baseline exists (with matching signature)? │ │ ├── YES → Run comparator → pass/fail │ │ │ ├── PASS → update baseline, push to HF │ │ │ └── FAIL → push diagnostics to HF │ │ └── NO → copy today's tensors as initial baseline │ │ → push to HF as "baseline_established" │ │ ↓ │ │ 9. Report summary (stdout + GitHub Step Summary) │ └──────────────────────────────────────────────────────────────┘ ``` ### Key components | Component | File | Purpose | | --------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------- | | Test entry point | `test/registered/debug_utils/test_nightly_precision_regression.py` | Orchestrates server launch, dump, compare, and reporting | | HF baseline store | `python/sglang/test/precision_baseline_store.py` | Push / fetch / prune baselines on a HuggingFace dataset | | Tensor comparator | `python/sglang/srt/debug_utils/comparator/` | Compares two directories of `.pt` tensors, emits JSONL report | | Dumper infrastructure | `python/sglang/srt/debug_utils/dumper.py` | Captures per-layer hidden states at runtime | | CI workflow | `.github/workflows/nightly-test-nvidia.yml` | Schedules the nightly job on 8×H200 | *** ## What Gets Dumped and Compared ### Strided layer capture Not every layer is dumped — the framework uses a **strided capture** to reduce I/O and storage overhead. By default, it captures: * Layer 0 (always) * The last layer (always) * Every 8th layer in between (configurable via `LAYER_CAPTURE_STRIDE`) The layer count is resolved automatically from the model's HuggingFace `config.json` (`num_hidden_layers` or `num_layers`). If resolution fails, all layers are captured as a safe fallback. The dumper filter is built dynamically as a regex matching only the selected layer indices, e.g.: ``` match(r'^non_intrusive__model\.layers\.(0|7|15|23)\.inputs\.1$', name) ``` ### Decode-path verification The test generates **2 tokens** with `ignore_eos=True` to ensure the model's decode path is exercised. After the dump, `_assert_decode_captured()` verifies that tensors from the decode step were actually captured (not just prefill). If only prefill tensors are found, the test fails immediately — this catches misconfigurations where `--max-total-tokens` is too low for the decode loop to run. ### Comparator The comparator computes **relative differences** (`rel_diff`) for each tensor and checks them against a configurable threshold (default `1e-3`). For tensor-parallel models, the `--override-dims` flag tells the comparator how to reduce across TP ranks before comparing: ``` --override-dims ^non_intrusive__model\.layers\.\d+\.inputs\.1$:bs h[tp:partial] ``` This sums partial TP contributions along the hidden dimension before computing the diff, so the comparison is semantically correct even with TP > 1. If the comparator returns exit code 0 but compared **zero layers** (baseline/target name mismatch), the test fails with a diagnostic message rather than silently passing. ### Capture signature A `capture_signature` (SHA-1 hash of schema version, max\_tokens, ignore\_eos, TP size, and dumper filter) is computed per run. The HF store uses this signature during fetch to ensure only baselines with an identical capture shape are considered. If the signature changes (e.g. you add layers to the capture set or change TP), the framework establishes a fresh baseline instead of erroring on incompatible tensors. *** ## Environment Variables | Variable | Default | Description | | --------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `SGLANG_PRECISION_MODELS` | `zai-org/GLM-5.1-FP8` | Comma-separated HuggingFace model IDs to test | | `SGLANG_PRECISION_BASELINE_DIR` | `/tmp/sglang_precision_baselines` | Local directory for baseline tensors | | `SGLANG_PRECISION_DIFF_THRESHOLD` | `1e-3` | Per-tensor relative diff threshold | | `SGLANG_PRECISION_FORCE_UPDATE` | `0` | Set to `1` to skip comparison and unconditionally refresh baseline | | `SGLANG_PRECISION_COMMIT` | *(auto-detected from git)* | Override the sglang commit SHA tagged on push | | `SGLANG_PRECISION_HF_REPO` | *(required)* | HuggingFace dataset repo for cross-runner baseline storage | | `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset | | `SGLANG_PRECISION_HF_TOKEN` | *(required in CI)* | HuggingFace token with write access to the dataset. Kept off `HF_TOKEN`, which already carries the runner's gated-model read token | *** ## CI Integration ### Workflow job The test is registered on the `nightly-8-gpu-h200` stage in `.github/workflows/nightly-test-nvidia.yml`, which runs through `_pr-test-stage.yml` like every other CUDA stage. The baseline env below is exported on every scheduled stage; only this test reads it. Key CI configuration: ```yaml theme={null} - name: Export precision baseline env if: inputs.scheduled env: BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} run: | { echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines" echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}" echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}" echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}" echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}" } >> "$GITHUB_ENV" ``` `SGLANG_PRECISION_HF_TOKEN` rather than `HF_TOKEN`: the latter already carries the runner's gated-model read token, and overwriting it would turn every gated model on the job into a 401. ### Required GitHub secrets/variables | Name | Type | Purpose | | ------------------------------ | ------------------------------ | --------------------------------------------------------------------------------------------------- | | `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset | | `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) | | `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset; exported to the job as `SGLANG_PRECISION_HF_TOKEN` | ### GitHub Step Summary When running in CI, the test writes a Markdown table to the GitHub Actions job summary showing each model's status (`PASSED`, `FAILED`, `BASELINE_ESTABLISHED`, or `ERROR`). *** ## HF Dataset Storage Layout Baselines are organized in the HF dataset as: ``` ///
/run-/ ├── meta.json # Run metadata (model, commit, hardware, thresholds, stats) ├── comparator_report.jsonl # Per-tensor comparison results └── tensors/ ├── layer_0_inputs_1.pt ├── layer_7_inputs_1.pt └── ... ``` A top-level `manifest.jsonl` tracks all runs with one JSON object per line. Each manifest row carries a `capture_signature` field so that fetch selects only baselines with a matching capture shape. The `prune_old_runs()` function (callable manually) retains daily runs for 30 days and keeps one run per week beyond that window. *** ## How to Add a New Model ### Option A: Add to the default model list (CI) Edit the default in `test/registered/debug_utils/test_nightly_precision_regression.py`: ```python theme={null} DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8,your-org/your-model" ``` Or set the `SGLANG_PRECISION_MODELS` environment variable in the CI workflow to override the default. ### Option B: Run locally for a specific model ```bash theme={null} export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines" export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` ### Step-by-step: adding a model to the nightly CI 1. **Verify the model works with the dumper.** Run locally first to ensure hidden states are captured correctly: ```bash theme={null} export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision ``` 2. **Run a comparison pass** (remove `FORCE_UPDATE`): ```bash theme={null} unset SGLANG_PRECISION_FORCE_UPDATE python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision ``` This should report `PASSED` if the engine is numerically stable for the model. 3. **Set the tensor-parallelism size.** If the model requires TP > 1, the test harness defaults to `tp_size=8` for all models. To customize, modify the `ModelLaunchSettings` construction in the test or pass extra server arguments: ```python theme={null} # In setUpClass or via env-driven logic cls.models = [ModelLaunchSettings("your-org/your-model", tp_size=4)] ``` 4. **Adjust the diff threshold if needed.** FP8 or quantized models may exhibit larger numerical differences. Set `SGLANG_PRECISION_DIFF_THRESHOLD` to an appropriate value (e.g., `1e-2` for FP8). 5. **Add to the default model list** or configure `SGLANG_PRECISION_MODELS` in the CI workflow. ### Considerations for model-specific adjustments | Concern | How to handle | | ------------------------------- | -------------------------------------------------------------------------------------------------------------- | | TP size != 8 | Override `tp_size` in `ModelLaunchSettings` or add model-specific logic | | Quantized models (FP8, GPTQ) | Loosen `SGLANG_PRECISION_DIFF_THRESHOLD` (e.g., `1e-2`) | | Model needs extra server args | Pass them via `ModelLaunchSettings(model, extra_args=["--quantization", "fp8"])` | | Model needs different prompt | Modify `PROMPT` constant or make it model-configurable | | MoE models with TP partial sums | Already handled by `--override-dims` (`bs h[tp:partial]`) | | Fewer/more capture layers | Adjust `LAYER_CAPTURE_STRIDE` (default 8); set lower for smaller models | | Decode not captured | Ensure `--max-total-tokens` is well above the scheduler's decode reservation (default 512); the test uses 4096 | *** ## Running Locally ### Prerequisites * SGLang installed in development mode * GPUs matching the model's requirements * `huggingface_hub` installed * A **HuggingFace dataset** for baseline storage and a write-capable `SGLANG_PRECISION_HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback. ### Quick local test ```bash theme={null} # All three are required — the test errors if SGLANG_PRECISION_HF_REPO is unset. export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct" export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_TOKEN="hf_..." # First run: establish baseline cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v # Second run: compare against baseline python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` ### Force-refresh a baseline ```bash theme={null} export SGLANG_PRECISION_FORCE_UPDATE="1" python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v ``` *** ## Interpreting Results ### Status codes | Status | Meaning | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `BASELINE_ESTABLISHED` | No prior baseline with a matching signature existed; today's tensors saved as the new baseline | | `PASSED` | All per-layer hidden states are within the diff threshold; baseline updated | | `FAILED` | One or more layers exceeded the diff threshold, or 0 layers were compared (baseline/target mismatch); diagnostic data pushed to HF | | `ERROR` | Server launch, inference, or comparison encountered an unexpected error | ### Output example ``` ============================================================ Nightly Precision Regression Summary ============================================================ Model Status Details ------------------------------------------------------------ zai-org/GLM-5.1-FP8 PASSED comparison ok, baseline updated Qwen/Qwen2.5-0.5B-Instruct FAILED tensor=layer_23.inputs_1 rel_diff=0.0152 ============================================================ ``` ### When a failure is detected 1. The comparator output is saved to `/tmp/nightly_precision__*.log` 2. The failing tensors and comparator report are pushed to the HF dataset with `pass_label="failed"` for offline diagnosis 3. The GitHub Step Summary includes the failure details 4. The CI job exits with a non-zero status *** ## Baseline Management ### Local baselines Baselines are stored at: ``` $SGLANG_PRECISION_BASELINE_DIR//nightly_precision/*.pt ``` A `baseline_meta.json` next to the tensors records the timestamp and commit that produced the baseline. ### HF dataset baselines * **Fetch:** At test start, if no local baseline exists, the latest signature-matched baseline is downloaded from the HF dataset. * **Push:** After each run, tensors and metadata are uploaded to the dataset. * **Prune:** Use `prune_old_runs()` to garbage-collect old baselines (keeps 30 days of daily runs, one per week after that). ### Refreshing a stale baseline If an intentional numerical change (e.g., kernel optimization, model refactor) causes a comparison failure: 1. Verify the change is intentional 2. Set `SGLANG_PRECISION_FORCE_UPDATE=1` and run the test once to establish a new baseline 3. Commit any necessary threshold adjustments If you change the capture configuration (stride, TP size, etc.), the `capture_signature` will differ and the framework automatically establishes a fresh baseline — no manual intervention needed. *** ## Known Limitations ### Baseline drift The framework uses a **rolling baseline**: every successful comparison updates the baseline to the current run's tensors. This means the reference shifts forward each day. While individual day-to-day diffs stay within the configured threshold, tiny numerical differences can **accumulate over time**, causing the baseline to silently drift away from the original golden values. **Implications:** * The framework detects **regressions** (a sudden, large numerical change between consecutive runs), not **absolute accuracy** relative to a fixed reference. * Over weeks or months, the cumulative drift may become significant enough to mask a real regression that happened gradually, or to cause a false-positive failure when the drift eventually crosses the threshold. **Mitigation strategies (not yet implemented):** * Periodically re-establish a fresh anchor baseline from a known-good reference commit. * Track the cumulative drift in the manifest metadata and alert when it exceeds a long-term budget. * Compare against a fixed "epoch" baseline in addition to the rolling one. ### No local-only mode The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `SGLANG_PRECISION_HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token. *** ## File Reference | File | Role | | ------------------------------------------------------------------ | ------------------------------------------------------------ | | `test/registered/debug_utils/test_nightly_precision_regression.py` | Main test — server lifecycle, dump, compare, report | | `python/sglang/test/precision_baseline_store.py` | HF dataset store — push, fetch, prune baselines | | `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine | | `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture | | `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition | | `test/run_suite.py` | Test suite registration (includes `nightly-test-8-gpu-h200`) | # References Source: https://docs.sglang.io/docs/references/overview FAQ, environment variables, production metrics, deployment guides, and more. * [FAQ](./faq) * [Environment Variables](./environment_variables) * [Production Metrics](./production_metrics) * [Production Request Trace](./production_request_trace) * [Multi-Node Deployment](./multi_node_deployment/multi_node) * [Custom Chat Template](./custom_chat_template) * [Frontend Language](./frontend/frontend_tutorial) * [Post-Training Integration](./post_training_integration) # Post-Training Integration Source: https://docs.sglang.io/docs/references/post_training_integration SGLang has become the de facto inference backend for modern LLM training frameworks, powering state-of-the-art models across the industry. From GLM-4.6 to Qwen3, leading models leverage SGLang's high-performance inference during reinforcement learning and post-training workflows. What makes SGLang essential for post-training? * Open-To-Use Refit Functionality: diverse method for colocate or disaggregate * Easy To Postpone Generation: enable partial rollout and dedicated rollout control * Fine-Grained Engine Sleep And Wake Up: facilitate maxium-powered rollout and training * Training Serving Alignment: ensure the performance consistency in training and serving * Load Balancing Router: cache-aware load-balancing for high-throughput rollout * Deterministic Inference: ensure zero kl divergence between rollout and training These capabilities, combined with native integration support across major frameworks, have established SGLang as the infrastructure backbone for modern LLM/VLMs post-training. We also share our latest work in this slide, [Optimizing Large-Scale RL with SGLang](https://gamma.app/docs/Optimizing-RL-with-SGLang-y0kqgj877k34779). ## Adoption * [**Miles**](https://github.com/radixark/miles): Enterprise-scale RL framework for large MoE models with SGLang-native rollout, speculative training, and production-grade stability * [**slime**](https://github.com/THUDM/slime): Post-training framework combining Megatron and SGLang, used to train GLM-4.6 * [**AReaL**](https://github.com/inclusionAI/AReaL): Fully asynchronous RL system achieving 2.77x speedup with SGLang backend for continuous rollout generation * [**ROLL**](https://github.com/alibaba/ROLL): ROLL is an efficient and user-friendly RL library designed for Large Language Models utilizing Large Scale GPU resources * [**verl**](https://github.com/volcengine/verl): Full-stack RLHF framework supporting PPO, GRPO, and ReMax with modular SGLang integration * [**Unsloth**](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide): 2x faster fine-tuning with optimized kernels, deploys seamlessly with SGLang inference * [**LLaMA Factory**](https://github.com/hiyouga/LLaMA-Factory): Unified framework for training 100+ LLMs with LoRA, QLoRA, and full fine-tuning methods * [**Tunix**](https://github.com/google/tunix): Google's JAX-native library for LLM post-training with SFT, DPO, PPO, and GRPO support * [**RL2**](https://github.com/ChenmienTan/RL2): Ray Less Reinforcement Learning, a concise library of post-training for large language models ## Collaboration Due to the privacy of the design parternes, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **[rl\_team@lmsys.org](mailto:rl_team@lmsys.org)** for partnerships, integration guidance, and custom feature development. # Production Metrics Source: https://docs.sglang.io/docs/references/production_metrics SGLang exposes the following metrics via Prometheus. You can enable it by adding `--enable-metrics` when you launch the server. An example of the monitoring dashboard is available in [examples/monitoring/grafana.json](https://github.com/sgl-project/sglang/blob/main/examples/monitoring/grafana/dashboards/json/sglang-dashboard.json). Here is an example of the metrics: ```text Output theme={null} $ curl http://localhost:30000/metrics # HELP sglang:prompt_tokens_total Number of prefill tokens processed. # TYPE sglang:prompt_tokens_total counter sglang:prompt_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.128902e+06 # HELP sglang:generation_tokens_total Number of generation tokens processed. # TYPE sglang:generation_tokens_total counter sglang:generation_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.557572e+06 # HELP sglang:token_usage The token usage # TYPE sglang:token_usage gauge sglang:token_usage{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.28 # HELP sglang:cache_hit_rate The cache hit rate # TYPE sglang:cache_hit_rate gauge sglang:cache_hit_rate{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.007507552643049313 # HELP sglang:time_to_first_token_seconds Histogram of time to first token in seconds. # TYPE sglang:time_to_first_token_seconds histogram sglang:time_to_first_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2.3518979474117756e+06 sglang:time_to_first_token_seconds_bucket{le="0.001",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:time_to_first_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0 sglang:time_to_first_token_seconds_bucket{le="0.06",model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0 sglang:time_to_first_token_seconds_bucket{le="0.08",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.25",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:time_to_first_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 27.0 sglang:time_to_first_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0 sglang:time_to_first_token_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 314.0 sglang:time_to_first_token_seconds_bucket{le="7.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 941.0 sglang:time_to_first_token_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1330.0 sglang:time_to_first_token_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1970.0 sglang:time_to_first_token_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2326.0 sglang:time_to_first_token_seconds_bucket{le="25.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2417.0 sglang:time_to_first_token_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2513.0 sglang:time_to_first_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0 sglang:time_to_first_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0 # HELP sglang:e2e_request_latency_seconds Histogram of End-to-end request latency in seconds # TYPE sglang:e2e_request_latency_seconds histogram sglang:e2e_request_latency_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.116093850019932e+06 sglang:e2e_request_latency_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 sglang:e2e_request_latency_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="0.8",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="1.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="2.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0 sglang:e2e_request_latency_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.0 sglang:e2e_request_latency_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 10.0 sglang:e2e_request_latency_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11.0 sglang:e2e_request_latency_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 14.0 sglang:e2e_request_latency_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 247.0 sglang:e2e_request_latency_seconds_bucket{le="40.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 486.0 sglang:e2e_request_latency_seconds_bucket{le="50.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 845.0 sglang:e2e_request_latency_seconds_bucket{le="60.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1513.0 sglang:e2e_request_latency_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0 sglang:e2e_request_latency_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0 # HELP sglang:time_per_output_token_seconds Histogram of time per output token in seconds. # TYPE sglang:time_per_output_token_seconds histogram sglang:time_per_output_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 866964.5791549598 sglang:time_per_output_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0 sglang:time_per_output_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 73.0 sglang:time_per_output_token_seconds_bucket{le="0.015",model_name="meta-llama/Llama-3.1-8B-Instruct"} 382.0 sglang:time_per_output_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 593.0 sglang:time_per_output_token_seconds_bucket{le="0.025",model_name="meta-llama/Llama-3.1-8B-Instruct"} 855.0 sglang:time_per_output_token_seconds_bucket{le="0.03",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1035.0 sglang:time_per_output_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1815.0 sglang:time_per_output_token_seconds_bucket{le="0.05",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11685.0 sglang:time_per_output_token_seconds_bucket{le="0.075",model_name="meta-llama/Llama-3.1-8B-Instruct"} 433413.0 sglang:time_per_output_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.950195e+06 sglang:time_per_output_token_seconds_bucket{le="0.15",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.039435e+06 sglang:time_per_output_token_seconds_bucket{le="0.2",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.171662e+06 sglang:time_per_output_token_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.266055e+06 sglang:time_per_output_token_seconds_bucket{le="0.4",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.296752e+06 sglang:time_per_output_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.312226e+06 sglang:time_per_output_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.339675e+06 sglang:time_per_output_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.357747e+06 sglang:time_per_output_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.389414e+06 sglang:time_per_output_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.400757e+06 sglang:time_per_output_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.400757e+06 # HELP sglang:func_latency_seconds Function latency in seconds # TYPE sglang:func_latency_seconds histogram sglang:func_latency_seconds_sum{name="generate_request"} 4.514771912145079 sglang:func_latency_seconds_bucket{le="0.05",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.07500000000000001",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.1125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.16875",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.253125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.3796875",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.56953125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="0.8542968750000001",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="1.2814453125",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="1.9221679687500002",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="2.8832519531250003",name="generate_request"} 14006.0 sglang:func_latency_seconds_bucket{le="4.3248779296875",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="6.487316894531251",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="9.730975341796876",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="14.596463012695313",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="21.89469451904297",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="32.84204177856446",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="49.26306266784668",name="generate_request"} 14007.0 sglang:func_latency_seconds_bucket{le="+Inf",name="generate_request"} 14007.0 sglang:func_latency_seconds_count{name="generate_request"} 14007.0 # HELP sglang:num_running_reqs The number of running requests # TYPE sglang:num_running_reqs gauge sglang:num_running_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 162.0 # HELP sglang:num_used_tokens The number of used tokens # TYPE sglang:num_used_tokens gauge sglang:num_used_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 123859.0 # HELP sglang:gen_throughput The generate throughput (token/s) # TYPE sglang:gen_throughput gauge sglang:gen_throughput{model_name="meta-llama/Llama-3.1-8B-Instruct"} 86.50814177726902 # HELP sglang:num_queue_reqs The number of requests in the waiting queue # TYPE sglang:num_queue_reqs gauge sglang:num_queue_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2826.0 # HELP sglang:spec_num_steps Currently active speculative_num_steps. # TYPE sglang:spec_num_steps gauge sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0 # HELP sglang:spec_num_draft_tokens Currently active speculative_num_draft_tokens (decouples from steps under topk>1). # TYPE sglang:spec_num_draft_tokens gauge sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0 ``` ## Setup Guide This section describes how to set up the monitoring stack (Prometheus + Grafana) provided in the `examples/monitoring` directory. ### Prerequisites * Docker and Docker Compose installed * SGLang server running with metrics enabled ### Usage 1. **Start your SGLang server with metrics enabled:** ```bash Command theme={null} python -m sglang.launch_server \ --model-path \ --port 30000 \ --enable-metrics \ --enable-mfu-metrics ``` Replace `` with the actual path to your model (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct`). Ensure the server is accessible from the monitoring stack (you might need `--host 0.0.0.0` if running in Docker). By default, the metrics endpoint will be available at `http://:30000/metrics`. 2. **Navigate to the monitoring example directory:** ```bash Command theme={null} cd examples/monitoring ``` 3. **Start the monitoring stack:** ```bash Command theme={null} docker compose up -d ``` This command will start Prometheus and Grafana in the background. 4. **Access the monitoring interfaces:** * **Grafana:** Open your web browser and go to [http://localhost:3000](http://localhost:3000). * **Prometheus:** Open your web browser and go to [http://localhost:9090](http://localhost:9090). 5. **Log in to Grafana:** * Default Username: `admin` * Default Password: `admin` You will be prompted to change the password upon your first login. 6. **View the Dashboard:** The SGLang dashboard is pre-configured and should be available automatically. Navigate to `Dashboards` -> `Browse` -> `SGLang Monitoring` folder -> `SGLang Dashboard`. ### Troubleshooting * **Port Conflicts:** If you encounter errors like "port is already allocated," check if other services (including previous instances of Prometheus/Grafana) are using ports `9090` or `3000`. Use `docker ps` to find running containers and `docker stop ` to stop them, or use `lsof -i :` to find other processes using the ports. You might need to adjust the ports in the `docker-compose.yaml` file if they permanently conflict with other essential services on your system. To modify Grafana's port to the other one(like 3090) in your Docker Compose file, you need to explicitly specify the port mapping under the grafana service. Option 1: Add GF\_SERVER\_HTTP\_PORT to the environment section: ``` environment: - GF_AUTH_ANONYMOUS_ENABLED=true - GF_SERVER_HTTP_PORT=3090 # <-- Add this line ``` Option 2: Use port mapping: ``` grafana: image: grafana/grafana:latest container_name: grafana ports: - "3090:3000" # <-- Host:Container port mapping ``` * **Connection Issues:** * Ensure both Prometheus and Grafana containers are running (`docker ps`). * Verify the Prometheus data source configuration in Grafana (usually auto-configured via `grafana/datasources/datasource.yaml`). Go to `Connections` -> `Data sources` -> `Prometheus`. The URL should point to the Prometheus service (e.g., `http://prometheus:9090`). * Confirm that your SGLang server is running and the metrics endpoint (`http://:30000/metrics`) is accessible *from the Prometheus container*. If SGLang is running on your host machine and Prometheus is in Docker, use `host.docker.internal` (on Docker Desktop) or your machine's network IP instead of `localhost` in the `prometheus.yaml` scrape configuration. * **No Data on Dashboard:** * Generate some traffic to your SGLang server to produce metrics. For example, run a benchmark: ```bash Command theme={null} python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompts 100 --random-input 128 --random-output 128 ``` * Check the Prometheus UI (`http://localhost:9090`) under `Status` -> `Targets` to see if the SGLang endpoint is being scraped successfully. * Verify the `model_name` and `instance` labels in your Prometheus metrics match the variables used in the Grafana dashboard. You might need to adjust the Grafana dashboard variables or the labels in your Prometheus configuration. ### Configuration Files The monitoring setup is defined by the following files within the `examples/monitoring` directory: * `docker-compose.yaml`: Defines the Prometheus and Grafana services. * `prometheus.yaml`: Prometheus configuration, including scrape targets. * `grafana/datasources/datasource.yaml`: Configures the Prometheus data source for Grafana. * `grafana/dashboards/config/dashboard.yaml`: Tells Grafana to load dashboards from the specified path. * `grafana/dashboards/json/sglang-dashboard.json`: The actual Grafana dashboard definition in JSON format. You can customize the setup by modifying these files. For instance, you might need to update the `static_configs` target in `prometheus.yaml` if your SGLang server runs on a different host or port. #### Check if the metrics are being collected Run: ```text Output theme={null} python3 -m sglang.bench_serving \ --backend sglang \ --dataset-name random \ --num-prompts 3000 \ --random-input 1024 \ --random-output 1024 \ --random-range-ratio 0.5 ``` to generate some requests. Then you should be able to see the metrics in the Grafana dashboard. ## Estimated Performance Metrics (MFU-related) SGLang exports the following estimated per-GPU counters that can be used to derive Model FLOPs Utilization (MFU)-related signals: * `sglang:estimated_flops_per_gpu_total`: Estimated floating-point operations. * `sglang:estimated_read_bytes_per_gpu_total`: Estimated bytes read from memory. * `sglang:estimated_write_bytes_per_gpu_total`: Estimated bytes written to memory. These metrics are available when both `--enable-metrics` and `--enable-mfu-metrics` are enabled. These are cumulative counters. Use Prometheus `rate(...)` to get per-second values. ### PromQL examples Average TFLOPS per GPU: ```promql theme={null} rate(sglang:estimated_flops_per_gpu_total[1m]) / 1e12 ``` Average estimated memory bandwidth in GB/s: ```promql theme={null} (rate(sglang:estimated_read_bytes_per_gpu_total[1m]) + rate(sglang:estimated_write_bytes_per_gpu_total[1m])) / 1e9 ``` ### Notes * These metrics are estimates intended for observability and trend analysis. * Estimated memory bytes reflect modeled traffic and are not a direct hardware counter from GPU profilers. # Production Request Tracing Source: https://docs.sglang.io/docs/references/production_request_trace SGLang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server. You can find example screenshots of the visualization in [https://github.com/sgl-project/sglang/issues/8965](https://github.com/sgl-project/sglang/issues/8965). ## Setup Guide This section explains how to configure the request tracing and export the trace data. 1. Install the required packages and tools * install Docker and Docker Compose * install the dependencies ```bash Command theme={null} # enter the SGLang root directory pip install -e "python[tracing]" # or manually install the dependencies using pip pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-grpc ``` 2. Launch OpenTelemetry collector and Jaeger ```bash Command theme={null} docker compose -f examples/monitoring/tracing_compose.yaml up -d ``` 3. Start your SGLang server with tracing enabled ```bash Command theme={null} # set env variables export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500 export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64 # start the prefill and decode server python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 # start the model-gate-way python -m sglang_router.launch_router --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` Replace `0.0.0.0:4317` with the actual endpoint of the OpenTelemetry collector. If you launched the openTelemetry collector with tracing\_compose.yaml, the default receiving port is 4317. To use the HTTP/protobuf span exporter, set the following environment variable and point to an HTTP endpoint, for example, `http://0.0.0.0:4318/v1/traces`. ```bash Command theme={null} export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf ``` 4. Raise some requests 5. Observe whether trace data is being exported * Access port 16686 of Jaeger using a web browser to visualize the request traces. * The OpenTelemetry Collector also exports trace data in JSON format to /tmp/otel\_trace.json. In a follow-up patch, we will provide a tool to convert this data into a Perfetto-compatible format, enabling visualization of requests in the Perfetto UI. 6. Dynamically adjust trace level The trace level accepts configurable values from `0` to `3`. The meanings of different trace level values are as follows: ``` 0: disable tracing 1: Trace important slices 2: Trace all slices except nested ones 3: Trace all slices (default) ``` **At startup** — set `SGLANG_TRACE_LEVEL` before launching the server: ```bash Command theme={null} SGLANG_TRACE_LEVEL=2 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` **At runtime** — dynamically adjust via HTTP API without restarting: ```bash Command theme={null} curl http://0.0.0.0:30000/set_trace_level?level=2 ``` Replace `0.0.0.0:30000` with your actual server address, and replace `level=2` with the level you want to set. **Note**: You must set the parameter `--enable-trace`; otherwise, the trace capability will not be enabled regardless of any dynamic adjustments to the trace level. ## Async Tracing (Reducing Performance Overhead) When batch sizes are large, synchronous OTel span creation can degrade inference throughput due to thread-safe locking and background export threads. Async tracing moves all span creation to a dedicated exporter process via ZMQ, keeping the scheduler and tokenizer hot paths free of OTel overhead. **Enable async tracing:** ```bash theme={null} SGLANG_TRACE_ASYNC=1 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 ``` **How it works:** * A daemon exporter process is started per worker process (scheduler, tokenizer, etc.). * The root span is still created in the caller process (one per request, negligible cost), preserving cross-process span linking via `traceparent`. * Thread spans and slice spans are buffered as lightweight operation dicts and flushed to the exporter via ZMQ PUSH/PULL. * Span IDs are pre-generated in the caller process using `TraceCustomIdGenerator.preset_next_span_id()`, so the exported span tree is identical to synchronous mode. * Thread info (scheduler label, TP/DP/PP ranks) is registered once via a callback on `trace_set_thread_info()`. **Tuning:** | Environment Variable | Description | Default | | ------------------------------------ | ---------------------------------- | ------- | | `SGLANG_TRACE_ASYNC` | Enable async tracing | `false` | | `SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD` | Max buffered ops before auto-flush | `100` | ## How to add Tracing for slices you're interested in?(API introduction) We have already inserted instrumentation points in the tokenizer and scheduler main threads. If you wish to trace additional request execution segments or perform finer-grained tracing, please use the APIs from the tracing package as described below. **All of the following implementations are done in python/sglang/srt/observability/req\_time\_stats.py. If you want to add another slice, please do it here.** 1. Initialization Every process involved in tracing during the initialization phase should execute: ```python Example theme={null} process_tracing_init(otlp_traces_endpoint, server_name) ``` The otlp\_traces\_endpoint is obtained from the arguments, and you can set server\_name freely, but it should remain consistent across all processes. Every thread involved in tracing during the initialization phase should execute: ```python Example theme={null} trace_set_thread_info("thread label", tp_rank, dp_rank) ``` The "thread label" can be regarded as the name of the thread, used to distinguish different threads in the visualization view. 2. Create a trace context for a request Each request needs to call `TraceReqContext()` to initialize a request context, which is used to generate slice spans and record request stage info. You can either store it within the request object or maintain it as a global variable. 3. Mark the beginning and end of a request ``` trace_ctx.trace_req_start(). trace_ctx.trace_req_finish() ``` trace\_req\_start() and trace\_req\_finish() must be called within the same process, for example, in the tokenizer. 4. Add tracing for a slice * Add slice tracing normally: ```python Example theme={null} trace_ctx.trace_slice_start(RequestStage.TOKENIZER.stage_name) trace_ctx.trace_slice_end(RequestStage.TOKENIZER.stage_name) or trace_ctx.trace_slice(slice: TraceSliceContext) ``` - The end of the last slice in a thread must be marked with thread\_finish\_flag=True, or explicitly call trace\_ctx.abort(); otherwise, the thread's span will not be properly generated. ```python Example theme={null} trace_ctx.slice_end(RequestStage.D.stage_name, thread_finish_flag = True) trace_ctx.abort() ``` 5. When the request execution flow transfers to another thread, the thread context needs to be explicitly rebuilt. * receiver: Execute the following code after receiving the request via ZMQ ```python Example theme={null} trace_ctx.rebuild_thread_context() ``` ## How to Extend the Tracing Framework to Support Complex Tracing Scenarios The currently provided tracing package still has potential for further development. If you wish to build more advanced features upon it, you must first understand its existing design principles. The core of the tracing framework's implementation lies in the design of the span structure and the trace context. To aggregate scattered slices and enable concurrent tracking of multiple requests, we have designed a three-level trace context structure or span structure: `TraceReqContext`, `TraceThreadContext` and `TraceSliceContext`. Their relationship is as follows: ``` TraceReqContext (req_id="req-123") ├── TraceThreadContext(thread_label="scheduler", tp_rank=0) | └── TraceSliceContext(slice_name="prefill") | └── TraceThreadContext(thread_label="scheduler", tp_rank=1) └── TraceSliceContext(slice_name="prefill") ``` Each traced request maintains a global `TraceReqContext` and creates a corresponding request span. For every thread that processes the request, a `TraceThreadContext` is recorded and a thread span is created. The `TraceThreadContext` is nested within the `TraceReqContext`, and each currently traced code slice—potentially nested—is stored in its associated `TraceThreadContext`. In addition to the above hierarchy, each slice also records its previous slice via Span.add\_link(), which can be used to trace the execution flow. # CLI reference Source: https://docs.sglang.io/docs/sglang-diffusion/api/cli Run one-off generation tasks and launch the HTTP server from the command line. Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`. ### Overlay repos for non-diffusers models If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it through a self-hosted overlay repo. SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface. Override example: ```bash Command theme={null} export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{ "Wan-AI/Wan2.2-S2V-14B": { "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay", "overlay_revision": "main" } }' sglang generate \ --model-path Wan-AI/Wan2.2-S2V-14B \ --config configs/wan_s2v.yaml ``` The overlay repo should be a complete diffusers-style/componentized repo You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`. Notes: 1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for development and debugging. It accepts either a JSON object or a path to a JSON file, and can extend or replace built-in entries for the current process. 2. On the first load, SGLang will: * download overlay metadata from the overlay repo * download the required files from the original source repo * materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/` 3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory. ## Quick Start ### Generate ```bash Command theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` ### Serve ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` For request and response examples, see [OpenAI-Compatible API](./openai_api). Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags. ## Common Options ### Model and runtime * `--model-path {MODEL}`: model path or Hugging Face model ID * `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`. * `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`. This is a server/load-time choice, unlike a request's `task`. * `--minimax-h3-adaln-cache-path {FILE}`: advanced MiniMax-H3-only inference cache. It replaces the checkpoint's AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant. * `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition. * `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter from a local path, Hugging Face repo/subfolder, or exact Hub file URL * `--lora-weight-name {FILE}`: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded. * `--lora-alpha {N}`: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and `adapter_config.json`. Do not set it when the adapter already records alpha metadata. * PEFT `adapter_config.json` semantics are applied automatically for named adapter slots, RSLoRA, and per-layer `alpha_pattern`. Adapters that require unsupported auxiliary parameters or runtime behavior, such as DoRA, fail before weight injection instead of silently using ordinary LoRA math. * `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. * `--num-gpus {N}`: number of GPUs to use * `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. `speed` keeps `torch.compile` disabled unless a model-specific deployment config opts in after validation; pass `--enable-torch-compile true` to enable it explicitly. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. * `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Startup impact is model-dependent, so benchmark the target model before deployment. Disabled by default because checkpoint and model weights coexist temporarily, substantially increasing peak GPU memory. It is incompatible with DiT CPU/layerwise offload and FSDP. * `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both. * `--sp-degree {N}`: sequence parallelism size * `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only. * `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls * `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading. * `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism * `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). * `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic * `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency. * `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP. * `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. A `quality=high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only high-quality paths remain compatible. * `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse. * `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines * `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa` * `--attention-backend-config {CONFIG}`: attention backend configuration * `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](../models_with_ar). * `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server * `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server * `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it. * `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](../models_with_pe). ### Sampling and output * `--prompt {PROMPT}` and `--negative-prompt {PROMPT}` * `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation * `--num-inference-steps {STEPS}` and `--seed {SEED}` * `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`. * `--quality {lossless,high}`: request-level quality. `lossless` (default) keeps the exact reference path, bit-exact against the reference implementation; `high` opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific. * `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` * `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` For frame interpolation and upscaling, see [Post-Processing](./post_processing). ### Quantization Component checkpoint paths are selected separately, so changing DiT precision never silently changes prompt embeddings. The component-scoped forms are the canonical interface for any component key from `model_index.json` or a native pipeline's registered module name: | Intent | Canonical option | Convenience alias | Behavior | | ------------------------------------------ | ------------------------------------------------------------------ | -------------------------------------- | ------------------------------------------------------------------- | | Replace a component | `--component-paths. {MODEL}` | `---path {MODEL}` | Load the replacement component's configuration and weights | | Replace only its weights | `--component-weights-paths. {WEIGHTS}` | `---weights-path {WEIGHTS}` | Retain the base component configuration and replace its weights | | Quantize an unquantized component online | `--component-quantizations. {METHOD}` | `---quantization {METHOD}` | Apply a method supported by that component's native loader | | Keep selected component layers unquantized | `--component-quantization-ignored-layers. {PATTERN...}` | None | Pass component-local ignored-layer patterns to its online quantizer | For example, pair a replacement text-encoder configuration with a separate single-file checkpoint as follows: ```bash theme={null} --component-paths.text_encoder Qwen/Qwen3-VL-4B-Instruct \ --component-weights-paths.text_encoder \ Comfy-Org/Krea-2/text_encoders/qwen3vl_4b_fp8_scaled.safetensors ``` The transformer-specific `--transformer-weights-path` spelling remains supported for the primary DiT. Do not mechanically replace it with `--component-weights-paths.transformer`: the component-scoped form requires an actual component name, which is pipeline-specific. Use `--quantization` only to override the method inferred by the transformer loader, and `--quantization-ignored-layers` to keep matching transformer layers unquantized during online quantization. For a native text encoder: * `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias * `--component-quantizations.text_encoder {METHOD}` applies supported online quantization to an unquantized native encoder; pair it with `--component-quantization-ignored-layers.text_encoder {PATTERN...}` when selected layers must remain unquantized * Quantization metadata is auto-detected from that checkpoint. The native loader accepts compatible serialized formats without a model-name allowlist and rejects implementations that do not construct the required quantized layers. The same contract applies to every weighted component: path routing is generic, while quantized materialization is capability-based. Native auxiliary loaders whose current materializer expects plain state dicts reject unsupported quantization metadata before model construction. See [Quantized Component Repositories](../quantization#quantized-component-repositories) for the current component matrix. Component overrides accept a local component directory, a standalone Hub repository, or a Hub component subfolder written as `owner/repo/subfolder`. For transformer and native encoder loaders, an explicit weight filename keeps the base component configuration and replaces only its weights; the selected file format must still be supported by that loader. For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}` compresses completed KV-cache chunks independently of transformer weight quantization. It is lossy and disabled by default. See [Realtime and Causal Video Models](../realtime_models) for the runtime and model scope, and [Quantization](../quantization) for supported quantization families and examples. ### Request logging * `--log-requests`: Log user-facing fields of all requests (default: `False`). The verbosity is decided by `--log-requests-level`. * `--log-requests-level {0|1|2|3}`: Verbosity level for request logging (default: `2`). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, ...). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt. * `--log-requests-format {text|json}`: Format for request logging (default: `text`). `text` is human-readable; `json` outputs structured JSON lines. * `--log-requests-target {TARGET...}`: Target(s) for request logging. Use `stdout` for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., `--log-requests-target stdout /my/log/dir`. ## Configuration Files Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file. ```bash Command theme={null} sglang generate --config config.yaml ``` Example: ```yaml Config theme={null} model_path: FastVideo/FastHunyuan-diffusers prompt: A beautiful woman in a red dress walking down a street output_path: outputs/ num_gpus: 2 sp_degree: 2 tp_size: 1 num_frames: 45 height: 720 width: 1280 num_inference_steps: 6 seed: 1024 fps: 24 precision: bf16 vae_precision: fp16 vae_tiling: true vae_sp: true enable_torch_compile: false ``` HunyuanVideo and FastHunyuan use tiled VAE decode by default so multi-GPU runs distribute VAE tiles instead of selecting spatial-shard decode. At HunyuanVideo's supported 960×544×77 shape, spatial-shard decode can consume 99.8 GiB per rank before requesting another 49.61 GiB causal mask. At FastHunyuan's default 1280×720×125 shape, the mask alone would require 197.75 GiB. You can still override the policy with `--vae-config.parallel-decode-mode`, but `spatial` and `spatial_shard` should only be used for smaller validated shapes. ## Generate `sglang generate` runs a single generation job and exits when the job finishes. ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --prompt "A curious raccoon" \ --save-output \ --output-path outputs \ --output-file-name "a-curious-raccoon.mp4" ``` HTTP server-only arguments are ignored by `sglang generate`. For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit). For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. LongCat-Image is supported by this path. Its DiT always consumes a fixed 512-token prompt body, so different prompt lengths reuse the same graph without model-specific padding. For a 1024x1024 deployment, use: ```bash Command theme={null} sglang serve --model-path meituan-longcat/LongCat-Image \ --enable-breakable-cuda-graph \ --warmup-resolutions 1024x1024 \ --enable-torch-compile false \ --port 30010 ``` SANA-Video supports the same path for fixed-shape serving. Its default text stage emits a fixed 300-token prompt shape, so the runtime reuses one graph across prompt lengths without padding it to a generic text bucket. BCG also requires the serving frame count to match warmup; the example captures the model's default 81-frame signature: ```bash Command theme={null} sglang serve \ --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \ --enable-breakable-cuda-graph \ --warmup-resolutions 832x480 \ --enable-torch-compile false \ --port 30010 ``` ### Component Residency Use `--component-residency COMPONENT=MODE` to assign one runtime residency mode to each native pipeline component: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \ --prompt "A quiet city street after rain" ``` The available modes are: * `resident`: keep the complete component on the accelerator. * `component-offload`: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward. * `layerwise-offload`: keep component weights on CPU and stream its declared layers during execution. Selectors match exact loaded component keys from `model_index.json`, including names such as `transformer_2`, `audio_vae`, and `connectors`. The group selectors `dit`, `text_encoder`, `image_encoder`, and `vae` are also available, together with `all`. An exact key overrides a matching group, and a group overrides `all`. Components without a matching canonical selector retain their explicit legacy setting or automatic/model default. The existing `--dit-cpu-offload`, `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, and `--cpu-offload-components` options remain supported. New and legacy options may be mixed: `--component-residency` wins only for components it matches, while unmatched legacy settings remain effective. Legacy layerwise selectors take precedence over legacy component-offload selectors for the same component. Explicit `--dit-layerwise-offload false` makes the DiT resident unless another explicit DiT selector, such as `--dit-cpu-offload true` or `--component-residency dit=component-offload`, selects a different mode. Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`. ### Layerwise Offload Tuning Use layerwise offload when a component does not fit comfortably in GPU memory. The compatibility options `--dit-layerwise-offload` and `--layerwise-offload-components` remain available (`--layerwise-offload-modules` is an alias), while new deployments can select the mode directly: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \ --dit-offload-prefetch-size 0 \ --prompt "A quiet city street after rain" ``` Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component. Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, and `--dit-layerwise-residency-policy` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. Those three set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON: ```bash theme={null} sglang serve --model-path \ --layerwise-offload-components dit,text_encoder \ --layerwise-prefetch-size text_encoder=2 \ --layerwise-resident-layers text_encoder=4 \ --layerwise-residency-policy text_encoder=strided ``` * `--layerwise-prefetch-size`: how many layers to fetch ahead. Fractional values are a share of the stack, `>= 1` an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers. * `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Resident layers are transferred once at startup, so they are removed from every pass. Fractional values are a share of the stack. * `--layerwise-residency-policy`: `leading` keeps the first layers, `strided` spreads them across the stack so the transfers do not arrive as one burst. A component without an entry keeps the group default, so adding these options changes nothing until one is set. Both knobs trade VRAM for transfer, and the return differs by component: a DiT earns it back once per denoising step, a text encoder or VAE once per request. Measure before raising either. ## Serve `sglang serve` starts the HTTP server and keeps the model loaded for repeated requests. ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` ### Health endpoints SGLang Diffusion separates process liveness from inference readiness: | Endpoint | Success condition | Recommended use | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `GET /liveness` | The HTTP server is accepting requests. It remains `200` during server warmup. | Kubernetes liveness probe | | `GET /health` | The server is ready for normal inference traffic. It returns `503` while server-based synthetic warmup is running and `200` after it completes. | Startup and readiness probes | | `GET /health_generate` | Compatibility alias for `/health`. It does not currently issue a generation request in SGLang Diffusion. | Existing integrations only | `/health` gates only server-based warmup. With `--warmup-mode off` or `--warmup-mode request`, it returns `200` once the HTTP server starts; those modes do not promise that compilation or other first-request work has completed. If server-based warmup fails, the server terminates instead of reporting ready. Do not use `/health` as a liveness probe: a long server warmup can legitimately keep it at `503` for several minutes. ### Cloud Storage SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation. ```bash Command theme={null} export SGLANG_CLOUD_STORAGE_TYPE=s3 export SGLANG_S3_BUCKET_NAME=my-bucket export SGLANG_S3_ACCESS_KEY_ID=your-access-key export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key export SGLANG_S3_ENDPOINT_URL=https://minio.example.com ``` See [Environment Variables](../environment_variables) for the full set of storage options. ## Component Path Overrides Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `---path`. ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.2-dev \ --vae-path fal/FLUX.2-Tiny-AutoEncoder ``` The component key must match a key in the model's `model_index.json` or a native pipeline's registered module name. The path must be either a Hugging Face repo ID or a complete component directory. Path selection and quantized checkpoint support are separate capabilities. A pre-quantized component repo must carry its quantization metadata, and the selected loader must support that serialized format. Native plain-state loaders fail closed; library-managed components inherit the corresponding Transformers or Diffusers support. The transformer-specific `--quantization` flag does not select the format of component checkpoints; their own metadata does. See [Quantized Component Repositories](../quantization#quantized-component-repositories). ## Component Attention Backend Overrides Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`. ```bash Command theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` only while that component is being constructed and otherwise fail if the component cannot satisfy them. Sparse self-attention backends use a compatible dense backend for cross-attention layers. The global backend remains strict for DiT components, while auxiliary components may fall back to a compatible backend. You can also pass dotted CLI entries: ```bash Command theme={null} sglang generate \ --model-path \ --component-attention-backends.text_encoder torch_sdpa \ --component-attention-backends.transformer fa ``` ## Diffusers Backend Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class. ### Key Options
Argument Values Description
--backend auto, sglang, diffusers Choose native SGLang, force native, or force diffusers
--attention-backend flash, \_flash\_3\_hub, sage, xformers, native Attention backend for diffusers pipelines
--trust-remote-code flag Required for models with custom pipeline classes
--vae-tiling and --vae-slicing flag Lower memory usage for VAE decode
--dit-precision and --vae-precision fp16, bf16, fp32 Precision controls
--enable-torch-compile flag Enable torch.compile
--cache-dit-config Cache-DiT config for diffusers pipelines
### Example ```bash theme={null} sglang generate \ --model-path AIDC-AI/Ovis-Image-7B \ --backend diffusers \ --trust-remote-code \ --attention-backend flash \ --prompt "A serene Japanese garden with cherry blossoms" \ --height 1024 \ --width 1024 \ --num-inference-steps 30 \ --save-output \ --output-path outputs \ --output-file-name ovis_garden.png ``` For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file. # OpenAI API Source: https://docs.sglang.io/docs/sglang-diffusion/api/openai_api Image and video generation endpoints with LoRA adapter management. The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management. ## Prerequisites * Python 3.11+ if you plan to use the OpenAI Python SDK. ## Serve Launch the server using the `sglang serve` command. ### Start the server ```bash theme={null} SERVER_ARGS=( --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers --served-model-name wan-t2v --text-encoder-cpu-offload --pin-cpu-memory --num-gpus 4 --ulysses-degree=2 --ring-degree=2 --port 30010 ) sglang serve "${SERVER_ARGS[@]}" ``` * **--model-path**: Path to the model or model ID. * **--served-model-name**: Stable model name exposed by the serving APIs. It defaults to `--model-id` when set, otherwise `--model-path`. * **--port**: HTTP port to listen on (default: `30000`). ### Served model name `--served-model-name` separates the public API identity from the checkpoint location. This is useful when replicas use different local mount paths or when a gateway needs one stable model name: ```bash theme={null} sglang serve \ --model-path /models/Wan2.1-T2V-1.3B-Diffusers \ --served-model-name wan-t2v \ --port 30010 ``` `--model-id` is not a free-form deployment alias: it selects the registered model configuration for checkpoints whose local path cannot be identified. `--served-model-name` only controls the name exposed by serving APIs. When both are set, the served name takes precedence for API responses. ### Discover the served model **Endpoint:** `GET /v1/models` Returns the public model name together with diffusion-specific runtime information. **Curl Example:** ```bash curl theme={null} curl -sS "http://localhost:30010/v1/models" ``` **Response Example:** ```json theme={null} { "object": "list", "data": [ { "id": "wan-t2v", "object": "model", "created": 1786348800, "owned_by": "sglang", "root": "wan-t2v", "parent": null, "max_model_len": null, "num_gpus": 4, "task_type": "T2V", "dit_precision": "bf16", "vae_precision": "fp16", "pipeline_name": "WanPipeline", "pipeline_class": "WanPipeline" } ] } ``` Retrieve the same model by its served name: ```bash curl theme={null} curl -sS "http://localhost:30010/v1/models/wan-t2v" ``` `GET /server_info` also reports `served_model_name` for gateway discovery. Video and action responses use this name when the request does not provide a model explicitly. *** ## Endpoints ### Image Generation The server implements an OpenAI-compatible Images API under the `/v1/images` namespace. **Create an image** **Endpoint:** `POST /v1/images/generations` #### Request quality `quality` selects a model-owned sampling level when that model advertises one: use `lossless` for the reference path or `high` for a validated accelerated path. Omit it (or send OpenAI's default `auto`) to keep the runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests. **Python Example (b64\_json response):** ```python Python theme={null} import base64 from openai import OpenAI client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1") img = client.images.generate( prompt="A calico cat playing a piano on stage", size="1024x1024", n=1, response_format="b64_json", ) image_bytes = base64.b64decode(img.data[0].b64_json) with open("output.png", "wb") as f: f.write(image_bytes) ``` **Curl Example:** ```bash curl theme={null} curl -sS -X POST "http://localhost:30010/v1/images/generations" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A calico cat playing a piano on stage", "size": "1024x1024", "n": 1, "response_format": "b64_json" }' ``` > **Note** > If `response_format=url` is used and cloud storage is not configured, the API returns > a relative URL like `/v1/images//content`. **Edit an image** **Endpoint:** `POST /v1/images/edits` This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image. **Curl Example (b64\_json response):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/images/edits" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "image=@local_input_image.png" \ -F "url=image_url.jpg" \ -F "prompt=A calico cat playing a piano on stage" \ -F "size=1024x1024" \ -F "response_format=b64_json" ``` **Curl Example (URL response):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/images/edits" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "image=@local_input_image.png" \ -F "url=image_url.jpg" \ -F "prompt=A calico cat playing a piano on stage" \ -F "size=1024x1024" \ -F "response_format=url" ``` **Download image content** When `response_format=url` is used with `POST /v1/images/generations` or `POST /v1/images/edits`, the API returns a relative URL like `/v1/images//content`. **Endpoint:** `GET /v1/images/{image_id}/content` **Curl Example:** ```bash theme={null} curl -sS -L "http://localhost:30010/v1/images//content" \ -H "Authorization: Bearer sk-proj-1234567890" \ -o output.png ``` ### Video Generation The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace. **Create a video (text-to-video)** **Endpoint:** `POST /v1/videos` **Python Example:** ```python Python theme={null} from openai import OpenAI client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1") video = client.videos.create( prompt="A calico cat playing a piano on stage", size="1280x720" ) print(f"Video ID: {video.id}, Status: {video.status}") ``` **Curl Example:** ```bash curl theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A calico cat playing a piano on stage", "size": "1280x720" }' ``` **Create a video (image-to-video)** For I2V or TI2V models (e.g., Wan2.1 I2V, LTX-2.3 two-stage), pass an input image via multipart form upload or a reference URL. **Curl Example (multipart form upload):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Authorization: Bearer sk-proj-1234567890" \ -F "prompt=A cat playing a piano" \ -F "input_reference=@input_image.png" \ -F "size=1280x720" ``` **Curl Example (reference URL):** ```bash Command theme={null} curl -sS -X POST "http://localhost:30010/v1/videos" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer sk-proj-1234567890" \ -d '{ "prompt": "A cat playing a piano", "reference_url": "https://example.com/input_image.png", "size": "1280x720" }' ``` **List videos** **Endpoint:** `GET /v1/videos` **Python Example:** ```python Python theme={null} videos = client.videos.list() for item in videos.data: print(item.id, item.status) ``` **Curl Example:** ```bash curl theme={null} curl -sS -X GET "http://localhost:30010/v1/videos" \ -H "Authorization: Bearer sk-proj-1234567890" ``` **Download video content** **Endpoint:** `GET /v1/videos/{video_id}/content` **Python Example:** ```python Python theme={null} import time # Poll for completion while True: page = client.videos.list() item = next((v for v in page.data if v.id == video_id), None) if item and item.status == "completed": break time.sleep(5) # Download content resp = client.videos.download_content(video_id=video_id) with open("output.mp4", "wb") as f: f.write(resp.read()) ``` **Curl Example:** ```bash curl theme={null} curl -sS -L "http://localhost:30010/v1/videos//content" \ -H "Authorization: Bearer sk-proj-1234567890" \ -o output.mp4 ``` *** ### LoRA Management The server supports dynamic loading, merging, and unmerging of LoRA adapters. **Important Notes:** * Mutual Exclusion: Only one LoRA configuration can be active per target at a time * Switching: To switch LoRAs, deactivate the current LoRA with `unmerge_lora_weights`, then `set` the new one * Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost **Set LoRA Adapter** Loads one or more LoRA adapters and applies them to the model. By default, regular weights are statically merged, while FSDP-sharded weights use dynamic LoRA to avoid full-gather memory peaks. **Endpoint:** `POST /v1/set_lora` **Parameters:** * `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs * `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname` * `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values: * `"all"` (default): Apply to all transformers * `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2) * `"transformer_2"`: Apply only to transformer\_2 (low noise for Wan2.2) * `"critic"`: Apply only to the critic model * `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect * `merge_mode` (string, optional): `"auto"` (default server policy), `"merge"` (force static merge), or `"dynamic"` (apply LoRA at forward time) **Single LoRA Example:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": "lora_name", "lora_path": "/path/to/lora.safetensors", "target": "all", "strength": 0.8 }' ``` **Multiple LoRA Example:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": ["lora_1", "lora_2"], "lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"], "target": ["transformer", "transformer_2"], "strength": [0.8, 1.0] }' ``` **Multiple LoRA with Same Target:** ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora \ -H "Content-Type: application/json" \ -d '{ "lora_nickname": ["style_lora", "character_lora"], "lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"], "target": "all", "strength": [0.7, 0.9] }' ``` When using multiple LoRAs: * All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length * If `target` or `strength` is a single value, it will be applied to all LoRAs * Multiple LoRAs applied to the same target are applied in order **Merge LoRA Weights** Manually merges the currently set LoRA weights into the base model. With FSDP-sharded weights, manual merge may require a full-gather and can OOM. Use `set_lora` with `merge_mode="auto"` or `"dynamic"` for the lower-peak path. **Endpoint:** `POST /v1/merge_lora_weights` **Parameters:** * `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer\_2", "critic" * `strength` (float, optional): LoRA strength for merge, default 1.0. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect **Curl Example:** ```bash theme={null} curl -X POST http://localhost:30010/v1/merge_lora_weights \ -H "Content-Type: application/json" \ -d '{"strength": 0.8}' ``` **Unmerge LoRA Weights** Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA. **Endpoint:** `POST /v1/unmerge_lora_weights` **Curl Example:** ```bash theme={null} curl -X POST http://localhost:30010/v1/unmerge_lora_weights \ -H "Content-Type: application/json" ``` **List LoRA Adapters** Returns loaded LoRA adapters and current application status per module. **Endpoint:** `GET /v1/list_loras` **Curl Example:** ```bash theme={null} curl -sS -X GET "http://localhost:30010/v1/list_loras" ``` **Response Example:** ```json theme={null} { "loaded_adapters": [ { "nickname": "lora_a", "path": "/weights/lora_a.safetensors" }, { "nickname": "lora_b", "path": "/weights/lora_b.safetensors" } ], "active": { "transformer": [ { "nickname": "lora2", "path": "tarn59/pixel_art_style_lora_z_image_turbo", "merged": true, "mode": "merged", "strength": 1.0 } ] } } ``` Notes: * If LoRA is not enabled for the current pipeline, the server will return an error. * `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter. ### Example: Switching LoRAs 1. Set LoRA A: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}' ``` 2. Generate with LoRA A... 3. Unmerge LoRA A: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/unmerge_lora_weights ``` 4. Set LoRA B: ```bash Command theme={null} curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}' ``` 5. Generate with LoRA B... ### Adjust Output Quality The server supports adjusting output quality and compression levels for both image and video generation through the `output-quality` and `output-compression` parameters. #### Parameters * **`output-quality`** (string, optional): Preset quality level that automatically sets compression. **Default is `"default"`**. Valid values: * `"maximum"`: Highest quality (100) * `"high"`: High quality (90) * `"medium"`: Medium quality (55) * `"low"`: Lower quality (35) * `"default"`: Auto-adjust based on media type (50 for video, 75 for image) * **`output-compression`** (integer, optional): Direct compression level override (0-100). **Default is `None`**. When provided (not `None`), takes precedence over `output-quality`. * `0`: Lowest quality, smallest file size * `100`: Highest quality, largest file size #### Notes * **Precedence**: When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence * **Format Support**: Quality settings apply to JPEG, and video formats. PNG uses lossless compression and ignores these settings * **File Size vs Quality**: Lower compression values (or "low" quality preset) produce smaller files but may show visible artifacts # Post-Processing Source: https://docs.sglang.io/docs/sglang-diffusion/api/post_processing SGLang diffusion supports optional post-processing steps that run after generation to improve temporal smoothness (frame interpolation) or spatial resolution (upscaling). These steps are independent of the diffusion model and can be combined in a single run. When both are enabled, **frame interpolation runs first** (increasing the frame count), then **upscaling runs on every frame** (increasing the spatial resolution). *** ## Frame Interpolation (video only) Frame interpolation synthesizes new frames between each pair of consecutive generated frames, producing smoother motion without re-running the diffusion model. The `--frame-interpolation-exp` flag controls how many rounds of interpolation to apply: each round inserts one new frame into every gap between adjacent frames, so the output frame count follows the formula: > **(N − 1) × 2^exp + 1** > > e.g. 5 original frames with `exp=1` → 4 gaps × 1 new frame + 5 originals = **9** frames; > with `exp=2` → **17** frames. ### CLI Arguments
Argument Description
--enable-frame-interpolation Enable frame interpolation. Model weights are downloaded automatically on first use.
--frame-interpolation-exp \{EXP} Interpolation exponent — 1 = 2× temporal resolution, 2 = 4×, etc. (default: 1)
--frame-interpolation-scale \{SCALE} RIFE inference scale; use 0.5 for high-resolution inputs to save memory (default: 1.0)
--frame-interpolation-model-path \{PATH} Local directory or HuggingFace repo ID containing RIFE flownet.pkl weights (default: elfgum/RIFE-4.22.lite, downloaded automatically)
### Supported Models Frame interpolation uses the [RIFE](https://github.com/hzwer/Practical-RIFE) (Real-Time Intermediate Flow Estimation) architecture. Only **RIFE 4.22.lite** (`IFNet` with 4-scale `IFBlock` backbone) is supported. The network topology is hard-coded, so custom weights provided via `--frame-interpolation-model-path` must be a `flownet.pkl` checkpoint that is compatible with this architecture. Other RIFE versions (e.g., older `v4.x` variants with different block counts) or entirely different frame interpolation methods (FILM, AMT, etc.) are **not supported**.
Weight HuggingFace Repo Description
RIFE 4.22.lite *(default)* elfgum/RIFE-4.22.lite Lightweight model, downloaded automatically on first use
### Example Generate a 5-frame video and interpolate to 9 frames ((5 − 1) × 2¹ + 1 = 9): ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --prompt "A dog running through a park" \ --num-frames 5 \ --enable-frame-interpolation \ --frame-interpolation-exp 1 \ --save-output ``` *** ## Upscaling (image and video) Upscaling increases the spatial resolution of generated images or video frames using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN). The model weights are downloaded automatically on first use and cached for subsequent runs. ### CLI Arguments
Argument Description
--enable-upscaling Enable post-generation upscaling using Real-ESRGAN.
--upscaling-scale \{SCALE} Desired upscaling factor (default: 4). The 4× model is used internally; if a different scale is requested, a bicubic resize is applied after the network output.
--upscaling-model-path \{PATH} Local .pth file, HuggingFace repo ID, or repo\_id:filename for Real-ESRGAN weights (default: ai-forever/Real-ESRGAN with RealESRGAN\_x4.pth, downloaded automatically). Use the repo\_id:filename format to specify a custom weight file from a HuggingFace repo (e.g. my-org/my-esrgan:weights.pth).
### Supported Models Upscaling supports two Real-ESRGAN network architectures. The correct architecture is **auto-detected** from the checkpoint keys, so you only need to point `--upscaling-model-path` at a valid `.pth` file:
Architecture Example Weights Description
RRDBNet RealESRGAN\_x4plus.pth Heavier model with higher quality; best for photos
SRVGGNetCompact RealESRGAN\_x4.pth *(default)*, realesr-animevideov3.pth, realesr-general-x4v3.pth Lightweight model; faster inference, good for video
The default weight file is [`ai-forever/Real-ESRGAN`](https://huggingface.co/ai-forever/Real-ESRGAN) with `RealESRGAN_x4.pth` (SRVGGNetCompact, 4× native scale). Other super-resolution models (e.g., SwinIR, HAT, BSRGAN) are **not supported** — only Real-ESRGAN checkpoints using the two architectures above are compatible. ### Examples Generate a 1024×1024 image and upscale to 4096×4096: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --prompt "A cat sitting on a windowsill" \ --output-size 1024x1024 \ --enable-upscaling \ --save-output ``` Generate a video and upscale each frame by 4×: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A curious raccoon" \ --enable-upscaling \ --upscaling-scale 4 \ --save-output ``` *** ## Combining Frame Interpolation and Upscaling Frame interpolation and upscaling can be combined in a single run. Interpolation is applied first (increasing the frame count), then upscaling is applied to every frame (increasing the spatial resolution). Example — generate 5 frames, interpolate to 9 frames, and upscale each frame by 4×: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A curious raccoon" \ --num-frames 5 \ --enable-frame-interpolation \ --frame-interpolation-exp 1 \ --enable-upscaling \ --upscaling-scale 4 \ --save-output ``` # Attention Backends Source: https://docs.sglang.io/docs/sglang-diffusion/attention_backends Select and configure attention backends for SGLang diffusion pipelines. This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them. ## Overview Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible component overrides fail unless a sparse backend is being replaced for cross-attention. When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). * **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. * **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. * **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. * **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. * **MPS**: always uses PyTorch SDPA. * **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. ## Backend options For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
CLI value Enum value Notes
`fa` / `fa3` / `fa4` `FA` FlashAttention. fa3/fa4 are normalized to fa during argument parsing (ServerArgs.**post\_init**).
`torch_sdpa` `TORCH_SDPA` PyTorch scaled\_dot\_product\_attention.
`sliding_tile_attn` `SLIDING_TILE_ATTN` Sliding Tile Attention (STA). Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` `SAGE_ATTN` Requires sageattention. On Hopper (SM90), PyPI sageattention==2.2.0 is unsupported because it lacks the upstream SM90 binding fix. Install pip install --force-reinstall git+[https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4](https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4) --no-build-isolation. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream setup.py.
`sage_attn_3` `SAGE_ATTN_3` Requires SageAttention3 installed per upstream instructions.
`sol_attn` `SOL_ATTN` Requires the upstream sol-attn package. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). BF16, head dim 128. Configure via --attention-backend-config.
`video_sparse_attn` `VIDEO_SPARSE_ATTN` Requires vsa. Configure sparsity via --attention-backend-config.
`vmoba_attn` `VMOBA_ATTN` Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
`aiter` `AITER` Requires aiter.
aiter\_sage AITER\_SAGE Requires aiter.
sla\_attn SLA\_ATTN Sparse Linear Attention. Requires SpargeAttn. Install with pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation.
sage\_sla\_attn SAGE\_SLA\_ATTN SageAttention + Sparse Linear Attention. Requires SpargeAttn (same install as SLA).
`sparse_video_gen_2_attn` `SPARSE_VIDEO_GEN_2_ATTN` Requires svg. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).
laser\_attn LASER\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
block\_sparse\_attn BLOCK\_SPARSE\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
rain\_fusion\_attn RAIN\_FUSION\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
## Selection priority The selection order in `runtime/layers/attention/selector.py` is: 1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)` 2. Component override from `--component-attention-backends` while that component is being constructed 3. CLI `--attention-backend` (`ServerArgs.attention_backend`) 4. Layer or component default, when declared 5. Auto selection (platform capability, dtype, and installed packages) An explicit global backend mismatch fails for DiT self-attention. Auxiliary components may fall back to their declared default or another compatible backend. Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: * A path to a JSON or YAML configuration file. * A JSON string (e.g., `'{"sparsity": 0.5}'`). * Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`). ### Supported Configuration Parameters **Sliding Tile Attention (`sliding_tile_attn`)**
Parameter Type Description Default
`mask_strategy_file_path` `str` **Required.** Path to the mask strategy JSON file. -
`sta_mode` `str` Mode of STA. `STA_inference`
`skip_time_steps` `int` Number of steps to use full attention before switching to sparse attention. `15`
**Video Sparse Attention (`video_sparse_attn`)**
Parameter Type Description Default
`sparsity` `float` Validation sparsity (0.0 - 1.0). `0.0`
**V-MoBA (`vmoba_attn`)**
Parameter Type Description Default
`temporal_chunk_size` `int` Chunk size for temporal dimension. -
`temporal_topk` `int` Top-K tokens to select in temporal dimension. -
`spatial_chunk_size` `list[int]` Chunk size for spatial dimension (H, W). -
`spatial_topk` `int` Top-K tokens to select in spatial dimension. -
`st_chunk_size` `list[int]` Chunk size for spatiotemporal dimension (T, H, W). -
`st_topk` `int` Top-K tokens to select in spatiotemporal dimension. -
`moba_select_mode` `str` Selection mode (e.g., `threshold`). `threshold`
`moba_threshold` `float` Threshold value for selection. `0.25`
`moba_threshold_type` `str` Type of thresholding (e.g., `query_head`). `query_head`
`first_full_step` `int` Number of initial steps to use full attention. `12`
`first_full_layer` `int` Number of initial layers to use full attention. `0`
`temporal_layer` `int` Number of temporal layers. `1`
`spatial_layer` `int` Number of spatial layers. `1`
`st_layer` `int` Number of spatiotemporal layers. `1`
**Block Sparse Attention (`block_sparse_attn`)**
Parameter Type Description Default
`skip_first_steps` `int` Number of steps to use laser attention before switching to sparse attention. `10`
`sparsity` `float` The sparsity coefficient must be in the range (0, 1). `0.2`
**Sol-Attn (`sol_attn`)**
Parameter Type Description Default
`tau` `float` Routing threshold scale. Higher values select fewer exact KV blocks. `1.0`
`thresh_type` `str` Threshold mode: `diag` or `exact`. `diag`
`sink_tokens` `int` Exact KV sink length for prefix tokens such as text/audio rows. `0`
`sink_start` `int` Start index of the exact KV sink range. `0`
`dense_steps` `int` Use dense attention for the first N denoising steps. `10`
`dense_layers` `str` Layer indices kept dense, e.g. `0,1` or `0-2`. `0,1`
`dense_backend` `str` Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate. `fa`
`kv_splits` `int | str` KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences. `auto`
## Platform support matrix
Backend CUDA ROCm XPU MUSA MPS NPU Notes
`fa` Yes Yes ✅ ✅ ❌ ✅ CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to torch\_sdpa. No extra installations are required for NPU
`torch_sdpa` Yes Yes Yes Yes ✅ ✅ Most compatible option across platforms.
`sliding_tile_attn` Yes No No No ❌ ❌ CUDA-only. Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` Yes No No Yes ❌ ❌ Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3` Yes No No No ❌ ❌ CUDA-only (optional dependency).
`sol_attn` Yes No No No ❌ ❌ CUDA-only. Requires sol-attn. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). Configure via --attention-backend-config.
`video_sparse_attn` Yes No No No ❌ ❌ CUDA-only. Requires vsa. Configure sparsity via --attention-backend-config.
sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
sage\_sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
vmoba\_attn Yes No No No ❌ ❌ CUDA-only. Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
aiter ❌ ✅ No No ❌ ❌ Requires aiter.
aiter\_sage ❌ ✅ No No ❌ ❌ Requires aiter.
`sparse_video_gen_2_attn` Yes No No No ❌ ❌ CUDA-only. Requires svg.
laser\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Uses SDPA if seqlen less than 2048.
block\_sparse\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Configuration via --attention-backend-config.
rain\_fusion\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu Configuration via --attention-backend-config.
## Usage ### Select a backend via CLI ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa ``` ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend torch_sdpa ``` ### Override one component Use component overrides when a specific module needs different attention semantics from the main transformer: ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. ### Per-request override (denoise loop) A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the `attention_backend_override` sampling param. Valid values are the exact/drop-in dense kernels — `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the dynamic-batch signature, so requests with different backends never share a batch. ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend-override sage_attn ``` ```python theme={null} client.images.generate( model="", prompt="...", extra_body={"attention_backend_override": "sage_attn"}, ) ``` Incompatible server settings **reject the request** (with a server log) instead of silently falling back: breakable CUDA graphs and `torch.compile` bake the attention kernel into a captured/traced graph; sparse server backends (`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set `dense_backend=sage_attn` to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on `torch_sdpa`: ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --attention-backend sol_attn \ --attention-backend-config dense_backend=sage_attn,dense_steps=10 \ --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`. ### Using Sliding Tile Attention (STA) ```bash theme={null} # Pass the mask strategy file path via config sglang generate \ --model-path \ --prompt "..." \ --attention-backend sliding_tile_attn \ --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json" ``` ### Notes for ROCm / MPS * ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. * MPS: the platform implementation always uses `torch_sdpa`. # Cache-DiT Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/cache_dit Configure Cache-DiT acceleration for diffusion inference. SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss. ## Overview **Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop: * **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences * **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions * **SCM (Step Computation Masking)**: Step-level caching control for additional speedup ## Basic Usage Cache-DiT is a **per-request** switch: each request decides whether to run cached or lossless, and requests with different Cache-DiT settings never share a batch. The `SGLANG_CACHE_DIT_*` environment variables remain available as server-wide defaults for requests that leave the switch unset. Enable it for a single generation: ```bash theme={null} sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --enable-cache-dit true ``` Or per request against a running server, via the OpenAI-compatible API: ```python theme={null} client.images.generate( model="Qwen/Qwen-Image", prompt="A beautiful sunset over the mountains", extra_body={ "enable_cache_dit": True, # optional knob overrides for this request only "cache_dit_params": {"residual_diff_threshold": 0.12, "scm_preset": "fast"}, }, ) ``` `enable_cache_dit` accepts three states: `true` (on for this request), `false` (off for this request, overriding the server default), and unset (follow the `SGLANG_CACHE_DIT_ENABLED` server default). `cache_dit_params` accepts the DBCache knobs (`Fn_compute_blocks`, `Bn_compute_blocks`, `max_warmup_steps`, `residual_diff_threshold`, `max_continuous_cached_steps`, `enable_taylorseer`, `taylorseer_order`), the SCM knobs (`scm_preset`, `scm_compute_bins`, `scm_cache_bins`, `scm_policy`), and a nested `secondary` dict with the DBCache knobs for the second transformer of dual-DiT models (unset secondary keys inherit the request's primary values, then the `SGLANG_CACHE_DIT_SECONDARY_*` defaults). To make Cache-DiT the default for every request instead, export the environment variable when launching: ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ sglang serve --model-path Qwen/Qwen-Image ``` ## Diffusers Backend Cache-DiT supports loading acceleration configs from a custom YAML file. For diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`). ### Single GPU inference Define a `cache.yaml` file that contains: * DBCache + TaylorSeer ```yaml theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 ``` Then apply the config with: ```bash theme={null} sglang generate \ --backend diffusers \ --model-path Qwen/Qwen-Image \ --cache-dit-config cache.yaml \ --prompt "A beautiful sunset over the mountains" ``` * DBCache + TaylorSeer + SCM (Step Computation Mask) ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 # Must set the num_inference_steps for SCM. The SCM will automatically # generate the steps computation mask based on the num_inference_steps. # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking num_inference_steps: 28 steps_computation_mask: fast ``` * DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 num_inference_steps: 28 steps_computation_mask: fast enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc. ``` ### Distributed inference * 1D Parallelism Define a parallelism only config yaml `parallel.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ``` Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses\_size. Otherwise, you should manually set it as specific int number, e.g, 4. Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference) ```bash theme={null} sglang generate \ --backend diffusers \ --num-gpus 4 \ --model-path Qwen/Qwen-Image \ --cache-dit-config parallel.yaml \ --prompt "A futuristic cityscape at sunset" ``` * 2D Parallelism You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto tp_size: 2 attention_backend: native ``` Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses\_size. * 3D Parallelism You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: 2 ring_size: 2 tp_size: 2 attention_backend: native ``` Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2. * Ulysses Anything Attention To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_anything: true ``` * Ulysses FP8 Communication For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_float8: true ``` * Async Ulysses CP You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image. ``` Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP. * TE-P and VAE-P You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains: ```yaml Config theme={null} parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] ``` ### Hybrid Cache and Parallelism Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains: ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] ``` Then, apply the hybrid cache and parallel acceleration config from yaml. ```bash theme={null} sglang generate \ --backend diffusers \ --num-gpus 4 \ --model-path Qwen/Qwen-Image \ --cache-dit-config hybrid.yaml \ --prompt "A beautiful sunset over the mountains" ``` ### Attention Backend In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains: ```yaml Config theme={null} attention_backend: "flash" # '_flash_3' for Hopper ``` ### Quantization You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains: ```yaml Config theme={null} quantize_config: # quantization configuration for transformer modules # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc. quant_type: "float8" # layers to exclude from quantization (transformer). layers that contains any of the # keywords in the exclude_layers list will be excluded from quantization. This is useful # for some sensitive layers that are not robust to quantization, e.g., embedding layers. exclude_layers: - "embedder" - "embed" verbose: false # whether to print verbose logs during quantization ``` Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example: ```bash Command theme={null} sglang generate \ --backend diffusers \ --model-path Qwen/Qwen-Image \ --warmup-mode request \ --cache-dit-config quantize.yaml \ --enable-torch-compile \ --dit-cpu-offload false \ --text-encoder-cpu-offload false \ --prompt "A beautiful sunset over the mountains" ``` ### Combined Configs: Cache + Parallelism + Quantization You can also combine all the above configs together in a single yaml file `combined.yaml` that contains: ```yaml Config theme={null} cache_config: max_warmup_steps: 8 warmup_interval: 2 max_cached_steps: -1 max_continuous_cached_steps: 2 Fn_compute_blocks: 1 Bn_compute_blocks: 0 residual_diff_threshold: 0.12 enable_taylorseer: true taylorseer_order: 1 parallelism_config: ulysses_size: auto attention_backend: native extra_parallel_modules: ["text_encoder", "vae"] quantize_config: quant_type: "float8" exclude_layers: - "embedder" - "embed" verbose: false ``` Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. ## Advanced Configuration ### DBCache Parameters DBCache controls block-level caching behavior:
Parameter Env Variable Default Description
Fn `SGLANG_CACHE_DIT_FN` 1 Number of first blocks to always compute
Bn `SGLANG_CACHE_DIT_BN` 0 Number of last blocks to always compute
W `SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching starts
R `SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
MC `SGLANG_CACHE_DIT_MC` 3 Maximum continuous cached steps
### TaylorSeer Configuration TaylorSeer improves caching accuracy using Taylor expansion:
Parameter Env Variable Default Description
Enable `SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
Order `SGLANG_CACHE_DIT_TS_ORDER` 1 Taylor expansion order (1 or 2)
### Combined Configuration Example DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters simultaneously: ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 \ SGLANG_CACHE_DIT_TAYLORSEER=true \ SGLANG_CACHE_DIT_TS_ORDER=2 \ sglang generate --model-path black-forest-labs/FLUX.1-dev \ --prompt "A curious raccoon in a forest" ``` ### SCM (Step Computation Masking) SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and which to use cached results. **SCM Presets** SCM is configured with presets:
Preset Compute Ratio Speed Quality
`none` 100% Baseline Best
`slow` \~75% \~1.3x High
`medium` \~50% \~2x Good
`fast` \~35% \~3x Acceptable
`ultra` \~25% \~4x Lower
**Usage** ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_PRESET=medium \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A futuristic cityscape at sunset" ``` **Custom SCM Bins** For fine-grained control over which steps to compute vs cache: ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \ SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A futuristic cityscape at sunset" ``` **SCM Policy**
Policy Env Variable Description
`dynamic` `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` Adaptive caching based on content (default)
`static` `SGLANG_CACHE_DIT_SCM_POLICY=static` Fixed caching pattern
## Environment Variables All Cache-DiT parameters can also be configured via environment variables, which act as the server-wide defaults for requests that don't set `enable_cache_dit` / `cache_dit_params`. See [Environment Variables](./environment_variables) for the complete list. ## Supported Models SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
Model Family Example Models
Wan Wan2.1, Wan2.2
Flux FLUX.1-dev, FLUX.2-dev
Z-Image Z-Image-Turbo
Qwen Qwen-Image, Qwen-Image-Edit
Hunyuan HunyuanVideo
MiniMax MiniMax-H3 (T2VA, FL2VA, and Ref2VA)
## Performance Tips 1. **Start with defaults**: The default parameters work well for most models 2. **Use TaylorSeer**: It typically improves both speed and quality 3. **Tune R threshold**: Lower values = better quality, higher values = faster 4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance 5. **Warmup matters**: Higher warmup = more stable caching decisions ## Limitations * **SGLang-native pipelines**: Distributed Cache-DiT paths exist for supported pipelines. Hybrid SP+TP configurations add communication and cache coordination overhead, so validate them on the target model and hardware before using them as production defaults. * **SCM minimum steps**: SCM requires >= 8 inference steps to be effective * **Model support**: The model must be registered in Cache-DiT's `BlockAdapterRegister` or have an SGLang custom block adapter. ## Troubleshooting ### SCM disabled for low step count For models with \< 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache acceleration still works. ## References * [Cache-DiT](https://github.com/vipshop/cache-dit) * [SGLang Diffusion](./performance-optimization) # Caching Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/caching-acceleration Compare caching acceleration strategies for diffusion models. SGLang provides three complementary caching strategies for Diffusion Transformer (DiT) models. All reduce denoising cost by skipping redundant computation, but they operate at different levels. ## Overview SGLang supports three complementary caching approaches:
Strategy Scope Mechanism Best For
Cache-DiT Block-level Skip individual transformer blocks dynamically Advanced, higher speedup
TeaCache Timestep-level Skip entire denoising steps based on L1 similarity Simple, built-in
Spectrum Timestep-level Forecast DiT features to skip selected denoising steps Experimental, model-validated tuning
## Cache-DiT [Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**. See [Cache-DiT](./cache_dit) for detailed configuration. Cache-DiT currently cannot be combined with `--use-fsdp-inference`. Keep FSDP disabled when enabling Cache-DiT, or use other residency/offload controls instead. ### Quick Start ```bash theme={null} SGLANG_CACHE_DIT_ENABLED=true \ sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" ``` ### Key Features * **DBCache**: Dynamic block-level caching based on residual differences * **TaylorSeer**: Taylor expansion-based calibration for optimized caching * **SCM**: Step-level computation masking for additional speedup ## TeaCache TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely. See [TeaCache](./teacache) for detailed documentation. ### Quick Overview * Tracks L1 distance between modulated inputs across timesteps * When accumulated distance is below threshold, reuses cached residual * Uses separate positive/negative caches for supported CFG model families ### Supported Models * Wan2.1 * Z-Image * Wan2.2: coefficients are not calibrated yet; enabling TeaCache is accepted but currently no-ops * HunyuanVideo: not supported yet For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled. ## Spectrum Spectrum forecasts DiT features and skips selected denoising steps. It is approximate and currently applies only to selected native implementation paths. See [Spectrum Acceleration](./spectrum) for supported model families, constraints, and request controls. ## References * [Cache-DiT Repository](https://github.com/vipshop/cache-dit) * [TeaCache Paper](https://arxiv.org/abs/2411.14324) # CI Performance Baselines Source: https://docs.sglang.io/docs/sglang-diffusion/ci_perf Generate and update diffusion performance baselines used in CI. ## Perf Baseline Generation Script `python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`. ### Usage Update a single case: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i ``` Select by regex: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*' ``` Run all keys from the baseline file `scenarios`: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline ``` Specify input/output paths and timeout: ```bash theme={null} python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600 ``` # Supported Models and Optimization Compatibility Source: https://docs.sglang.io/docs/sglang-diffusion/compatibility_matrix Check supported SGLang Diffusion models and their optimization compatibility. This page tracks supported SGLang Diffusion model families and their optimization compatibility. It also covers long-tail models that do not yet have dedicated cookbook recipes. For model-specific usage recipes, start from the [Diffusion Cookbook](/cookbook/diffusion/intro). Cookbook pages cover the primary models with examples; this page keeps the compact support and compatibility inventory. ## Supported model inventory Pass the `Hugging Face Model ID` to `--model-path` for `sglang generate` or `sglang serve`. Python API users can pass the same ID to SGLang Diffusion model-loading helpers. Missing checkpoint aliases do not imply that a model family is unsupported. The runtime registry may also accept detector-based aliases or local model directories that match the same family. Rows are grouped when a family shares the same runtime path or optimization support. Use the detailed matrix below when you need per-optimization compatibility.
Model family Model IDs
FLUX
black-forest-labs/FLUX.1-devblack-forest-labs/FLUX.2-devblack-forest-labs/FLUX.2-dev-NVFP4black-forest-labs/FLUX.2-klein-4Bblack-forest-labs/FLUX.2-klein-9Bblack-forest-labs/FLUX.2-klein-base-4Bblack-forest-labs/FLUX.2-klein-base-9B
Z-Image
Tongyi-MAI/Z-ImageTongyi-MAI/Z-Image-Turbo
Qwen-Image
Qwen/Qwen-ImageQwen/Qwen-Image-2512Qwen/Qwen-Image-EditQwen/Qwen-Image-Edit-2509Qwen/Qwen-Image-Edit-2511Qwen/Qwen-Image-Layered
LongCat-Image
meituan-longcat/LongCat-Imagemeituan-longcat/LongCat-Image-Editmeituan-longcat/LongCat-Image-Edit-Turbo
SD3 / SD3.5
stabilityai/stable-diffusion-3-mediumstabilityai/stable-diffusion-3-medium-diffusersstabilityai/stable-diffusion-3.5-mediumstabilityai/stable-diffusion-3.5-medium-diffusersstabilityai/stable-diffusion-3.5-largestabilityai/stable-diffusion-3.5-large-diffusers
SANA
Efficient-Large-Model/SANA1.5\_1.6B\_1024px\_diffusersEfficient-Large-Model/SANA1.5\_4.8B\_1024px\_diffusersEfficient-Large-Model/Sana\_1600M\_1024px\_diffusersEfficient-Large-Model/Sana\_600M\_1024px\_diffusersEfficient-Large-Model/Sana\_1600M\_512px\_diffusersEfficient-Large-Model/Sana\_600M\_512px\_diffusers
FireRed-Image
FireRedTeam/FireRed-Image-Edit-1.0FireRedTeam/FireRed-Image-Edit-1.1
JoyAI-Image
jdopensource/JoyAI-Image-Edit-Diffusers
Other image pipelines
zai-org/GLM-Imagetencent/Hunyuan3D-2baidu/ERNIE-Imagebaidu/ERNIE-Image-Turboideogram-ai/ideogram-4-fp8ideogram-ai/ideogram-4-nf4Comfy-Org/Ideogram-4fal/ideogram-v4-fastfal/ideogram-v4-instant
Model family Model IDs Resolution / mode Optimization support
FastWan
FastVideo/FastWan2.1-T2V-1.3B-DiffusersFastVideo/FastWan2.2-TI2V-5B-FullAttn-DiffusersFastVideo/FastWan2.2-TI2V-5B-Diffusers
 480p / 720p VSA
SANA-Video
Efficient-Large-Model/SANA-Video\_2B\_480p\_diffusers
 T2V, 480p No dedicated optimization listed
LingBot Video MoE
robbyant/lingbot-video-moe-30b-a3b
 T2V, 480p No dedicated optimization listed
Wan2.2
Wan-AI/Wan2.2-TI2V-5B-DiffusersWan-AI/Wan2.2-T2V-A14B-Diffusersnvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4Wan-AI/Wan2.2-I2V-A14B-Diffusers
 TI2V / T2V / I2V, 480p / 720p SageLaserBSARain Fusion
LongLive 2.0
Rabinovich/LongLive-2.0-5B-Diffusers
 T2V / I2V, 480p / 720p No dedicated optimization listed
HunyuanVideo
hunyuanvideo-community/HunyuanVideoFastVideo/FastHunyuan-diffusers
 720×1280 / 544×960 TileSageSVG2
Wan2.1
Wan-AI/Wan2.1-T2V-1.3B-DiffusersWan-AI/Wan2.1-T2V-14B-DiffusersWan-AI/Wan2.1-I2V-14B-480P-DiffusersWan-AI/Wan2.1-I2V-14B-720P-Diffusers
 T2V / I2V, 480p / 720p TeaCacheTileSageSVG2LaserBSARain Fusion
TurboWan
IPostYellow/TurboWan2.1-T2V-1.3B-DiffusersIPostYellow/TurboWan2.1-T2V-14B-DiffusersIPostYellow/TurboWan2.1-T2V-14B-720P-DiffusersIPostYellow/TurboWan2.2-I2V-A14B-Diffusers
 480p / 720p TeaCacheSLASageSLA
MOVA
OpenMOSS-Team/MOVA-360pOpenMOSS-Team/MOVA-720p
 Video-audio, 360p / 720p; local MOVA detector aliases are also supported. No dedicated optimization listed
MiniMax-H3
MiniMaxAI/MiniMax-H3
 T2VA / FL2VA / Ref2VA, 768p at 24 fps with synchronized audio Cache-DiTSageOnline FP8GGUF
Wan2.1 Fun
weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers
 480p inpainting TeaCacheTileSageSVG2
Helios
BestWishYsh/Helios-BaseBestWishYsh/Helios-MidBestWishYsh/Helios-Distilled
 720p No dedicated optimization listed
LTX-2
Lightricks/LTX-2Lightricks/LTX-2.3
 One-stage, two-stage, TI2V, HQ No dedicated optimization listed
LTX-2.5
Lightricks/LTX-2.5-Diffusers
 One-stage, two-stage, TI2V, auto-duration, diffusion decode No dedicated optimization listed
Cosmos3
nvidia/Cosmos3-Nanonvidia/Cosmos3-Supernvidia/Cosmos3-Super-Text2Imagenvidia/Cosmos3-Super-Image2Video
 T2V / I2V / T2I No dedicated optimization listed
Model family Model IDs / detector Notes
LingBotWorld
robbyant/lingbot-world-fast-diffusers
 Realtime world model with causal state and control tokens.
SANA-WM
Efficient-Large-Model/SANA-WM\_bidirectionalEfficient-Large-Model/SANA-WM\_streaming
 World-model pipeline with bidirectional and streaming checkpoints.
Wan2.2 TI2V 5B currently has known quality issues when used for I2V generation. ## Optimization compatibility The detailed video matrix uses these symbols: * ✅ = Full compatibility * ❌ = No compatibility * ⭕ = Does not apply to this model ### Video Generation Models Optimization columns are abbreviated to keep the matrix readable: * `Tea` = TeaCache * `Tile` = Sliding Tile Attention * `Sage` = Sage Attention * `VSA` = Video Sparse Attention * `SLA` = Sparse Linear Attention * `SageSLA` = Sage Sparse Linear Attention * `SVG2` = Sparse Video Gen 2 * `LA` = Laser Attention * `BSA` = Block Sparse Attention * `RF` = Rain Fusion Attention
Model Name Hugging Face Model ID Resolution Tea Tile Sage VSA SLA SageSLA SVG2 LA BSA RF
FastWan2.1 T2V 1.3B `FastVideo/FastWan2.1-T2V-1.3B-Diffusers` 480p ⭕ ⭕ ⭕ ✅ ❌ ❌ ❌ ❌ ❌ ❌
SANA-Video 2B Efficient-Large-Model/SANA-Video\_2B\_480p\_diffusers 480p ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
LingBot Video MoE 30B-A3B robbyant/lingbot-video-moe-30b-a3b 480p ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
FastWan2.2 TI2V 5B FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers
FastVideo/FastWan2.2-TI2V-5B-Diffusers		 720p ⭕ ⭕ ⭕ ✅ ❌ ❌ ❌ ❌ ❌ ❌
Wan2.2 TI2V 5B `Wan-AI/Wan2.2-TI2V-5B-Diffusers` 720p ⭕ ⭕ ✅ ⭕ ❌ ❌ ❌ ✅ ✅ ✅
LongLive 2.0 5B Rabinovich/LongLive-2.0-5B-Diffusers 480p
720p		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
Wan2.2 T2V A14B Wan-AI/Wan2.2-T2V-A14B-Diffusers
nvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4		 480p
720p		 ❌ ❌ ✅ ⭕ ❌ ❌ ❌ ✅ ✅ ✅
Wan2.2 I2V A14B `Wan-AI/Wan2.2-I2V-A14B-Diffusers` 480p
720p		 ❌ ❌ ✅ ⭕ ❌ ❌ ❌ ✅ ✅ ✅
HunyuanVideo `hunyuanvideo-community/HunyuanVideo` 720×1280
544×960		 ❌ ✅ ✅ ⭕ ❌ ❌ ✅ ❌ ❌ ❌
FastHunyuan `FastVideo/FastHunyuan-diffusers` 720×1280
544×960		 ❌ ✅ ✅ ⭕ ❌ ❌ ✅ ❌ ❌ ❌
Wan2.1 T2V 1.3B `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` 480p ✅ ✅ ✅ ⭕ ❌ ❌ ✅ ✅ ✅ ✅
Wan2.1 T2V 14B `Wan-AI/Wan2.1-T2V-14B-Diffusers` 480p, 720p ✅ ✅ ✅ ⭕ ❌ ❌ ✅ ✅ ✅ ✅
Wan2.1 I2V 480P `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` 480p ✅ ✅ ✅ ⭕ ❌ ❌ ✅ ✅ ✅ ✅
Wan2.1 I2V 720P `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` 720p ✅ ✅ ✅ ⭕ ❌ ❌ ✅ ✅ ✅ ✅
TurboWan2.1 T2V 1.3B `IPostYellow/TurboWan2.1-T2V-1.3B-Diffusers` 480p ✅ ❌ ❌ ❌ ✅ ✅ ⭕ ❌ ❌ ❌
TurboWan2.1 T2V 14B `IPostYellow/TurboWan2.1-T2V-14B-Diffusers` 480p ✅ ❌ ❌ ❌ ✅ ✅ ⭕ ❌ ❌ ❌
TurboWan2.1 T2V 14B 720P `IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers` 720p ✅ ❌ ❌ ❌ ✅ ✅ ⭕ ❌ ❌ ❌
TurboWan2.2 I2V A14B `IPostYellow/TurboWan2.2-I2V-A14B-Diffusers` 720p ✅ ❌ ❌ ❌ ✅ ✅ ⭕ ❌ ❌ ❌
Wan2.1 Fun 1.3B InP weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers 480p ✅ ✅ ✅ ⭕ ❌ ❌ ✅ ❌ ❌ ❌
Helios Base BestWishYsh/Helios-Base 720p ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
Helios Mid BestWishYsh/Helios-Mid 720p ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
Helios Distilled BestWishYsh/Helios-Distilled 720p ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
LTX-2 (one/two-stage/TI2V) Lightricks/LTX-2 768×512
1536×1024		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
MiniMax-H3 (T2VA / FL2VA / Ref2VA image, audio, video/V2V) MiniMaxAI/MiniMax-H3 768p · 24 fps ❌ ❌ ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌
LTX-2.3 (one/two-stage/TI2V/HQ) Lightricks/LTX-2.3 768×512
1536×1024
1920×1088 (HQ default)		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
LTX-2.5 (one/two-stage/TI2V) Lightricks/LTX-2.5-Diffusers 960×544 (default)
1920×1088 (two-stage)		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
Cosmos3-Nano (T2V / I2V / T2I) nvidia/Cosmos3-Nano 720p · 480p
1024×1024 (T2I)		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
Cosmos3-Super (T2V / I2V / T2I) nvidia/Cosmos3-Super
nvidia/Cosmos3-Super-Text2Image
nvidia/Cosmos3-Super-Image2Video		 720p · 480p
1024×1024 (T2I)		 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌
**Note**: 1. Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue. 2. SageSLA is based on SpargeAttn. Install it first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation` 3. LTX pipeline selection: * One-stage: `--pipeline-class-name LTX2Pipeline` * Two-stage: `--pipeline-class-name LTX2TwoStagePipeline` * Two-stage HQ: `--pipeline-class-name LTX2TwoStageHQPipeline` (HQ defaults to 1920×1088; you can still override `--width/--height`) * LTX-2 and LTX-2.3 support both T2V and TI2V (`--image-path`) on one-stage and two-stage pipelines (including HQ). * The spatial upsampler and distilled LoRA are auto-resolved from the model snapshot by default, and can still be overridden with `--spatial-upsampler-path` and `--distilled-lora-path`. * For LTX models, the `Resolutions` column uses output video `width×height` semantics, matching `sglang generate --width ... --height ...`. 4. LTX-2 / LTX-2.3 two-stage also supports `--ltx2-two-stage-device-mode {original,resident}`: * `original` keeps official two-stage semantics without the premerged stage-2 transformer path. * `resident` usually provides the best latency/throughput but uses much more VRAM. * Default is auto: `resident` on H200/high-memory CUDA GPUs, otherwise `original`. 5. Cosmos3 ships in two sizes — `nvidia/Cosmos3-Nano` (16B) and `nvidia/Cosmos3-Super` (64B). Both share the same pipeline; the only difference is transformer depth and width, picked up from `transformer/config.json` at load time. A single checkpoint serves T2V, I2V (`--image-path`), and T2I (`--num-frames 1`). ## Supported Components SGLang Diffusion supports overriding individual pipeline components with `---path`. The value can be either a Hugging Face repo ID or a local component directory. The same overrides can also be provided in config files through `component_paths.`. Component path overrides are generic, but quantized checkpoint support is loader-specific. Native SGLang loaders that only materialize plain state dicts reject detected quantization metadata before weight loading instead of silently casting the checkpoint into an unquantized module. Library-managed components inherit the corresponding Transformers or Diffusers support. See [Quantized Component Repositories](./quantization#quantized-component-repositories). ### Common Syntax CLI: ```bash Command theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --vae-path black-forest-labs/FLUX.2-small-decoder \ --transformer-path /models/flux2/transformer ``` Config file: ```yaml Config theme={null} model_path: black-forest-labs/FLUX.2-dev component_paths: vae: black-forest-labs/FLUX.2-small-decoder transformer: /models/flux2/transformer ``` Use the component name from the pipeline's `model_index.json` or the native pipeline's registered module name:
Component Type Supported Keys Notes
VAE vae, video\_vae, audio\_vae vae is the common image-generation override
Transformer / DiT transformer, video\_dit, audio\_dit transformer is the standard override for the main denoiser
Text / Preprocess text\_encoder, text\_encoder\_2, tokenizer, processor, image\_processor Replacement encoders often need matching preprocessing assets
Auxiliary scheduler, spatial\_upsampler, vocoder, connectors, dual\_tower\_bridge, image\_encoder, vision\_language\_encoder Only valid for pipelines that expose these components
### Known Component Repos The table below lists concrete Hugging Face component repos that are already used in SGLang Diffusion docs or tests. It is not an exhaustive catalog of all compatible component repos.
Base Model Override Key Example Repo Notes
black-forest-labs/FLUX.2-dev vae black-forest-labs/FLUX.2-small-decoder Decoder-only FLUX.2 VAE override
black-forest-labs/FLUX.2-dev vae fal/FLUX.2-Tiny-AutoEncoder Existing tested custom VAE path
### VAE * `--vae-path` is the common image-generation override. * `--video-vae-path` and `--audio-vae-path` are only relevant for pipelines with separate video or audio VAEs. ### Transformer / DiT * `--transformer-path` is the standard override for the main denoising transformer. * For a pre-quantized main transformer, prefer `--transformer-path` or `--transformer-weights-path`; see [Quantization](./quantization). * `--video-dit-path` and `--audio-dit-path` are only for pipelines that split denoisers by modality. ### Text Encoders and Preprocessors * `--text-encoder-path` and `--text-encoder-2-path` override primary and secondary text encoders. * `--tokenizer-path`, `--processor-path`, and `--image-processor-path` are useful when the replacement encoder requires matching preprocessing assets. ### Auxiliary Components * `--scheduler-path` is only relevant when the pipeline exposes a scheduler component. * `--spatial-upsampler-path` is mainly for two-stage pipelines such as `LTX2TwoStagePipeline`. * `--vocoder-path`, `--connectors-path`, `--dual-tower-bridge-path`, `--image-encoder-path`, and `--vision-language-encoder-path` are only valid for pipelines that expose those components. ### Notes 1. Component overrides are only valid when the target pipeline actually uses that component. 2. The override key should match the component name in the pipeline's `model_index.json` or the native pipeline's registered module name. 3. An override path does not enable quantization by itself. A pre-quantized component must include metadata for a format supported by its selected loader. `--quantization` overrides the transformer loader; other component checkpoints select their serialized format through their own metadata. ## Verified LoRA Examples This section lists example LoRAs that have been explicitly tested and verified with each base model in the **SGLang Diffusion** pipeline. LoRAs that are not listed here are not necessarily incompatible. In practice, most standard LoRAs are expected to work, especially those following common Diffusers or SD-style conventions. The entries below simply reflect configurations that have been manually validated by the SGLang team. ### Verified LoRAs by Base Model
Base Model Supported LoRAs
MiniMax-H3 `larryvrh/MiniMax-H3-Turbo-Lora`
`lightx2v/Minimax-h3-Turbo`
`fal/MiniMax-H3-Realism-People-LoRA`
Wan2.2 `lightx2v/Wan2.2-Distill-Loras`
`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1`
Wan2.1 `lightx2v/Wan2.1-Distill-Loras`
Z-Image-Turbo `tarn59/pixel_art_style_lora_z_image_turbo`
`wcde/Z-Image-Turbo-DeJPEG-Lora`
Qwen-Image `lightx2v/Qwen-Image-Lightning`
`flymy-ai/qwen-image-realism-lora`
`prithivMLmods/Qwen-Image-HeadshotX`
`starsfriday/Qwen-Image-EVA-LoRA`
Qwen-Image-Edit `ostris/qwen_image_edit_inpainting`
`lightx2v/Qwen-Image-Edit-2511-Lightning`
Flux `dvyio/flux-lora-simple-illustration`
`XLabs-AI/flux-furry-lora`
`XLabs-AI/flux-RealismLora`
## Special requirements ### Sliding Tile Attention * Currently, only Hopper GPUs (H100s) are supported. # Contributing to SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/contributing This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`). ## Contributor Guides * [Support New Models](./support_new_models): implementation guide for adding new diffusion pipelines * [CI Performance](./ci_perf): update and regenerate perf baselines ## On AI-Assisted ("Vibe Coding") PRs Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs: * **No over-commenting.** If the name says it all, skip the docstring. * **No over-catching.** Don't guard against errors that virtually never happen in practice. * **Test before submitting.** AI-generated code can be subtly wrong — verify correctness end-to-end. ## Commit Message Convention We follow a structured commit message format to maintain a clean history. **Format:** ```text theme={null} [diffusion] : ``` **Examples:** * `[diffusion] cli: add --perf-dump-path argument` * `[diffusion] scheduler: fix deadlock in batch processing` * `[diffusion] model: support Stable Diffusion 3.5` **Rules:** * **Prefix**: Always start with `[diffusion]`. * **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc. * **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature"). ## Performance Reporting For PRs that impact **latency**, **throughput**, or **memory usage**, you **should** provide a performance comparison report. ### How to Generate a Report 1. **Baseline**: run the benchmark (for a single generation task) ```bash theme={null} $ sglang generate --model-path --prompt "A benchmark prompt" --perf-dump-path baseline.json ``` 2. **New**: run the same benchmark, without modifying any server\_args or sampling\_params ```bash theme={null} $ sglang generate --model-path --prompt "A benchmark prompt" --perf-dump-path new.json ``` 3. **Compare**: run the compare script, which will print a Markdown table to the console ```bash theme={null} $ python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json [new2.json ...] ### Performance Comparison Report ... ``` 4. **Paste**: paste the table into the PR description ## CI-Based Change Protection Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard your changes, especially for PRs that: * support a new model * add a testcase for this new model to `testcase_configs.py` * support or fix important features * significantly improve performance Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable. See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples # Deployment and Performance Modes Source: https://docs.sglang.io/docs/sglang-diffusion/deployment_cookbook Choose component residency, FSDP, CFG parallelism, SP, TP, and performance-mode presets in SGLang Diffusion. This page gives practical defaults for choosing `--performance-mode`, component residency, FSDP, CFG parallelism, SP, and TP. ## Quick Rule Use the simplest setting that fits your memory target:
Goal Recommended setting
Fastest single-GPU run when the model fits Use resident components and do not use FSDP.
Lower single-GPU memory usage Use component offload, then layerwise offload when a complete component still does not fit comfortably.
Faster multi-GPU Qwen/Wan CFG generation Use FSDP with CFG parallelism and keep the sharded component resident.
Sequence length or video-shape scaling Use SP/Ulysses/Ring when the model benefits from sequence parallelism.
TP compatibility or encoder-heavy paths Set TP explicitly; do not treat TP as the default latency optimization.
Base the decision on available memory on the selected GPU(s). * For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU. * For single-GPU deployment: FSDP shards weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; use component or layerwise offload instead. ## Health Probes Use `/liveness` to check that the HTTP process is alive and `/health` to check that the server is ready for inference. During server-based warmup, `/liveness` returns `200` while `/health` returns `503`. Configure the startup probe with a failure budget large enough for model loading and compilation: ```yaml theme={null} startupProbe: httpGet: path: /health port: 30010 periodSeconds: 10 failureThreshold: 180 readinessProbe: httpGet: path: /health port: 30010 livenessProbe: httpGet: path: /liveness port: 30010 ``` See [Health endpoints](/docs/sglang-diffusion/api/cli#health-endpoints) for the status-code contract and warmup-mode behavior. ## Stable Model Identity Use `--served-model-name` when the public model name must remain stable across replicas, hosts, or checkpoint mount paths: ```bash theme={null} sglang serve \ --model-path /mnt/checkpoints/Qwen-Image \ --model-id Qwen-Image \ --served-model-name image-production \ --port 30010 ``` The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path`. `--model-id` remains an internal model registry and configuration-resolution hint; it is not a replacement for a deployment alias. The resolved name is exposed through `/server_info` and `/v1/models` and is used by video and action responses when a request does not supply its own model. See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served-model-name) for discovery and retrieval examples. ## Performance Modes `--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.
Mode Meaning
`manual` Keeps performance-related server args under explicit user control.
`auto` Default. Keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload.
`speed` Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset. `torch.compile` stays off unless the model has a validated default or it is enabled explicitly; may OOM.
`memory` Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.
`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Model-specific components such as VAEs become resident only when their configured memory threshold is met. `memory` instead keeps the VAE in its default layerwise set to maximize memory headroom; use `--component-residency vae=resident` for a measured recipe with sufficient capacity. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly. The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode. For direct control, assign one of `resident`, `component-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency dit=layerwise-offload text_encoder=component-offload vae=resident ``` Existing per-component CPU-offload and layerwise flags remain supported. Canonical selectors override matching legacy settings only; unmatched legacy settings and automatic defaults remain effective. See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency) for the complete precedence rules. When `torch.compile` is enabled, `--offload-during-compile` stays on by default. During compile warmup it temporarily offloads the DiT and evicts resident non-DiT components so `max-autotune` fits on tighter-memory GPUs, then restores the configured serving residency before real traffic. Breakable CUDA graph is a separate manual opt-in for supported image pipelines. If you enable `--enable-breakable-cuda-graph`, declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides. Examples: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto ``` ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --performance-mode memory ``` Explicit flags win over the mode: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --num-gpus 2 \ --performance-mode auto \ --use-fsdp-inference false ``` In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled. ## Interpreting The Levers **Resident** keeps the complete component on the accelerator. It is usually fastest when memory is sufficient. **Component offload** keeps a complete component on CPU between declared uses. It is simple and robust, but each use pays a whole-component transfer. **Layerwise offload** streams the declared layers of any supported native weighted component. It lowers peak accelerator memory further, but may increase latency and lower throughput. **FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads. FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths. **CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far. **SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency. **TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan. ## Current Benchmark Takeaways Observed regular-scale trends: * Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it. * Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware. * Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path. * Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs. Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults. # Disaggregated Diffusion Pipeline Source: https://docs.sglang.io/docs/sglang-diffusion/disaggregation Split a monolithic text-to-video/image pipeline into independent **Encoder**, **Denoiser**, and **Decoder** roles, each running on its own GPU(s). A central **DiffusionServer** routes requests through the pipeline. ## Quick Start Disaggregation is controlled by a single flag: `--disagg-role`. Each component is launched independently, just like LLM PD disaggregation.
--disagg-role What it runs
monolithic (Default) Standard single-server mode
encoder All stages with the default RoleType.ENCODER affinity: InputValidationStage, TextEncodingStage (plus ImageEncodingStage / ImageVAEEncodingStage for image-conditioned pipelines), LatentPreparationStage, TimestepPreparationStage, and any model-specific "before denoising" stage (e.g. QwenImageLayeredBeforeDenoisingStage, GlmImageBeforeDenoisingStage).
denoiser DenoisingStage (and its subclasses: CausalDMDDenoisingStage, DmdDenoisingStage, LTX2AVDenoisingStage, LTX2RefinementStage, Hunyuan3DShapeDenoisingStage, ...) — the DiT forward loop plus the scheduler stepping it drives.
decoder DecodingStage (VAE decode) and its subclasses (LTX2AVDecodingStage, HeliosDecodingStage, ...).
server DiffusionServer head node + HTTP server (no GPU)
> Each stage declares its role via the `role_affinity` property on `PipelineStage` (default `ENCODER`). When `--disagg-role` is not `monolithic`, the pipeline only instantiates stages whose affinity matches, so the above table is the source of truth for what actually runs in each process. ### Single-Machine Example (Verified) The following commands have been tested end-to-end on an 8×H200 machine with `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`. Each role runs on a separate GPU via `--base-gpu-id`; the `server` head node requires no GPU. ```bash theme={null} # Terminal 1: Encoder (GPU 0) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role encoder \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19000 \ --num-gpus 1 --base-gpu-id 0 # Terminal 2: Denoiser (GPU 1) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role denoiser \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19001 \ --num-gpus 1 --base-gpu-id 1 # Terminal 3: Decoder (GPU 2) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role decoder \ --disagg-server-addr tcp://127.0.0.1:19655 \ --scheduler-port 19002 \ --num-gpus 1 --base-gpu-id 2 # Terminal 4: DiffusionServer head (no GPU, receives HTTP requests) sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --disagg-role server \ --encoder-urls "tcp://127.0.0.1:19000" \ --denoiser-urls "tcp://127.0.0.1:19001" \ --decoder-urls "tcp://127.0.0.1:19002" \ --host 0.0.0.0 --port 22000 \ --scheduler-port 19655 # Send request (video generation) curl http://127.0.0.1:22000/v1/videos \ -H "Content-Type: application/json" \ -d '{"model": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", "prompt": "A curious raccoon exploring a garden, cinematic", "size": "832x480"}' ``` > **Tested result (8×H200):** > Encoder 2.3 s (TextEncoding) → Denoiser 312.8 s (50 steps, layerwise offload) → Decoder 7.1 s (VAE decode). > Total \~322 s for 81-frame 1024×1024 video. > **Tip:** `--base-gpu-id` controls which physical GPU the role uses. > Encoder and Decoder can share a GPU (e.g. both `--base-gpu-id 0`) to save resources, > but make sure the combined GPU memory is sufficient. ### Multi-Machine Example The exact same CLI pattern — just replace `127.0.0.1` with actual IPs and add RDMA flags for direct transfer: ```bash theme={null} # Machine A (10.0.0.1): Encoder sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role encoder \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19000 \ --num-gpus 1 \ --disagg-p2p-hostname 10.0.0.1 --disagg-ib-device mlx5_0 # Machine B (10.0.0.2): Denoiser (4 GPUs with SP) sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role denoiser \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19001 \ --num-gpus 4 --denoiser-sp 4 --denoiser-ulysses 2 --denoiser-ring 2 \ --disagg-p2p-hostname 10.0.0.2 --disagg-ib-device mlx5_0 # Machine C (10.0.0.3): Decoder sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role decoder \ --disagg-server-addr tcp://10.0.0.4:19655 \ --scheduler-port 19002 \ --num-gpus 1 \ --disagg-p2p-hostname 10.0.0.3 --disagg-ib-device mlx5_0 # Machine D (10.0.0.4): DiffusionServer head sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \ --disagg-role server \ --encoder-urls "tcp://10.0.0.1:19000" \ --denoiser-urls "tcp://10.0.0.2:19001" \ --decoder-urls "tcp://10.0.0.3:19002" \ --host 0.0.0.0 --port 30000 \ --scheduler-port 19655 \ --disagg-dispatch-policy max_free_slots ``` > ZMQ handles startup order gracefully — instances and head can start in any order. ## Multiple Instances per Role Use semicolons in `--*-urls` to register multiple instances: ```bash theme={null} # 2 encoders + 2 denoisers (4-GPU SP each) + 1 decoder sglang serve --model-path ... --disagg-role server \ --encoder-urls "tcp://10.0.0.1:35000;tcp://10.0.0.2:35000" \ --denoiser-urls "tcp://10.0.0.3:35000;tcp://10.0.0.4:35000" \ --decoder-urls "tcp://10.0.0.5:35000" ``` ## Port Convention Result endpoints are derived deterministically from the head node's `--scheduler-port` (default: 5555):
Socket Port
DS frontend (ROUTER) scheduler\_port
Encoder result (PULL) scheduler\_port + 1
Denoiser result (PULL) scheduler\_port + 2
Decoder result (PULL) scheduler\_port + 3
Role instances derive their result endpoint automatically from `--disagg-server-addr`. No manual endpoint configuration needed. ## Transfer Mechanism Tensor data between roles (encoder→denoiser, denoiser→decoder) is transferred via a P2P transfer engine. The DiffusionServer only routes lightweight control messages (alloc/push/ready); actual tensor data flows directly between instances. **mooncake-transfer-engine** is required for disaggregated diffusion. It provides RDMA for direct GPU-to-GPU data movement. ```bash theme={null} pip install mooncake-transfer-engine ``` ### Transfer Flow 1. **Sender** (encoder/denoiser) stages tensors: async copy to transfer buffer (GPU or CPU pinned, depending on GPUDirect support), overlapped with metadata JSON serialization. 2. **Sender** sends `transfer_staged` control message to DiffusionServer (metadata only, no tensor data). 3. **DiffusionServer** sends `transfer_alloc` to receiver → receiver allocates buffer slot → replies `transfer_allocated`. 4. **DiffusionServer** sends `transfer_push` to receiver with sender's address info. 5. **Receiver** pulls data via transfer engine (Mooncake RDMA or mock), sends `transfer_ready`. 6. **Receiver** loads tensors async on a dedicated transfer stream, overlapped with the previous request's compute. Decoder results (final output) flow back through DiffusionServer as raw ZMQ frames to the HTTP client. ### RDMA Flags
Flag Default Description
--disagg-p2p-hostname 127.0.0.1 RDMA-reachable hostname/IP of this instance
--disagg-ib-device None InfiniBand device (e.g., mlx5\_0, mlx5\_roce0)
--disagg-transfer-pool-size 256 MiB Pinned memory pool per instance
Set `--disagg-p2p-hostname` to the actual IP on each machine. For multi-machine, `--disagg-ib-device` specifies the RDMA NIC. ## Per-Role Parallelism
Flag Description
--encoder-tp Encoder tensor parallelism
--denoiser-tp / --denoiser-sp / --denoiser-ulysses / --denoiser-ring Denoiser parallelism
--decoder-sp Decoder sequence parallelism
If not specified, parallelism is auto-derived from `--num-gpus`. ## Other Options
Flag Default Description
--disagg-timeout 600 Timeout (seconds) for pending requests
--disagg-dispatch-policy round\_robin round\_robin or max\_free\_slots
## Python API For programmatic single-machine deployment, `launch_pool_disagg_server()` is available: ```python theme={null} from sglang.multimodal_gen.runtime.server_args import ServerArgs from sglang.multimodal_gen.runtime.launch_server import launch_pool_disagg_server server_args = ServerArgs.from_kwargs( model_path="Wan-AI/Wan2.1-T2V-14B-Diffusers", denoiser_sp=4, denoiser_ulysses=2, denoiser_ring=2, disagg_ib_device="mlx5_0", ) launch_pool_disagg_server( server_args, encoder_gpus=[[0]], denoiser_gpus=[[1, 2, 3, 4], [5, 6, 7, 8]], decoder_gpus=[[0]], ) ``` ## Architecture ``` Client ─── HTTP (port 30000) ──► FastAPI Server │ ▼ DiffusionServer (ROUTER, scheduler_port) ┌───────┼───────┐ PUSH work │ │ │ PUSH work ▼ │ ▼ Encoder[0..N] │ Decoder[0..K] │ │ ▲ P2P tensor │ │ │ P2P tensor transfer ▼ │ │ transfer Denoiser[0..M] ─────┘ │ PULL results ◄────┘ (decoder → DS → client) ``` ### Request State Machine ``` PENDING → ENCODER_WAITING → ENCODER_RUNNING → ENCODER_DONE │ DENOISING_WAITING → DENOISING_RUNNING → DENOISING_DONE │ DECODER_WAITING → DECODER_RUNNING → DONE ``` Any state can transition to `FAILED` or `TIMED_OUT`. # Inference Batching Source: https://docs.sglang.io/docs/sglang-diffusion/dynamic_batching Batch compatible native SGLang-Diffusion requests during serving. Dynamic batching is an opt-in SGLang-Diffusion serving mode that merges compatible queued requests into one native pipeline batch. It is separate from LLM continuous batching and tokenizer batching. Use it for concurrent T2I or T2V traffic with the same model and sampling shape. Keep singleton serving for latency-sensitive or highly mixed traffic. ## Enable Dynamic batching is disabled by default with `--batching-max-size 1`. For GLM-Image T2I with an external AR server, compatible requests share one batched AR `/generate` call. DiT denoising and VAE decoding remain independent per-request executions; GLM-Image does not batch DiT work. ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.1-dev \ --port 30010 \ --batching-mode dynamic \ --batching-max-size 8 \ --batching-delay-ms 5 \ --enable-batching-metrics ``` For request formats, see the [OpenAI-Compatible API](./api/openai_api). Use `--batching-config /path/to/batching_config.json` to load JSON rules when a model or resolution needs a lower cap than `--batching-max-size`: ```json Config theme={null} { "schema_version": 1, "rules": [ { "model_contains": "Qwen-Image", "resolution": "1024x1024", "max_batch_size": 1 } ] } ``` ## Compatibility An initial implementation of dynamic batching for T2I and T2V models can be found in [#18764](https://github.com/sgl-project/sglang/pull/18764). The current compatibility grid is below and will be updated as more coverage is added. See [Supported Models and Optimization Compatibility](./compatibility_matrix) for common model IDs and optimization support. `✅` means supported, `❌` means not currently supported, `?` means untested, and `-` means not applicable. ### Image
Model T2I I2I
FLUX.1-dev✅-
FLUX.2-dev✅❌
FLUX.2-dev-NVFP4??
FLUX.2-Klein-4B✅❌
FLUX.2-Klein-9B??
FLUX.2-Klein-Base-4B??
FLUX.2-Klein-Base-9B??
Z-Image?-
Z-Image-Turbo✅-
GLM-Image (external AR)✅-
Qwen Image✅-
Qwen Image 2512✅-
Qwen Image Edit-❌
Qwen Image Edit 2509-?
Qwen Image Edit 2511-?
Qwen Image Layered??
SD3 Medium?-
SD3.5 Medium?-
SD3.5 Large?-
Hunyuan3D-2?-
SANA 1.5 1.6B✅-
SANA 1.5 4.8B✅-
SANA 1600M 1024px?-
SANA 600M 1024px?-
SANA 1600M 512px?-
SANA 600M 512px?-
FireRed-Image-Edit 1.0-?
FireRed-Image-Edit 1.1-?
ERNIE-Image?-
ERNIE-Image-Turbo?-
### Video
Model Support
FastWan2.1 T2V 1.3B✅
FastWan2.2 TI2V 5B Full Attn❌
Wan2.2 TI2V 5B❌
Wan2.2 T2V A14B✅
Wan2.2 I2V A14B❌
HunyuanVideo❌
FastHunyuan❌
Wan2.1 T2V 1.3B✅
Wan2.1 T2V 14B✅
Wan2.1 I2V 480P?
Wan2.1 I2V 720P?
TurboWan2.1 T2V 1.3B✅
TurboWan2.1 T2V 14B✅
TurboWan2.1 T2V 14B 720P✅
TurboWan2.2 I2V A14B?
Wan2.1 Fun 1.3B InP?
Helios Base?
Helios Mid?
Helios Distilled?
LTX-2?
LTX-2.3?
## Notes * Requests batch only when model inputs, sampling parameters, output handling, and any configured rules are compatible. * There is no startup probing, runtime learning, OOM retry, or automatic fallback to singletons. If a merged batch fails or cannot be split, every request in that batch receives an error. * Batch shape can change kernels, so singleton and dynamic outputs are not expected to be bit-exact. * Use `--enable-batching-metrics` to inspect realized batches: ```text theme={null} Dynamic batch dispatch: size=2/8, user_max=8, queue_wait=5.12ms, stop_reason=delay Dynamic batch dispatch: size=1/8, user_max=8, queue_wait=0.04ms, stop_reason=config_cap:1 Dynamic batch stats (last 5 dispatches): avg_size=2.80, merged_rate=60.0%, full_rate=20.0%, utilization=35.0%, wait_avg=3.21ms, wait_p95=5.12ms, top_rejects=none ``` # Encoder Parallelism Source: https://docs.sglang.io/docs/sglang-diffusion/encoder_parallel While the DiT denoises, the text and image encoders are idle — and while they encode, the whole DiT replica is idle. `--encoder-parallel` decides how to use those otherwise-unused GPUs for the encoding stage. ```bash theme={null} --encoder-parallel {auto,fold,dp,replicate} ``` | Mode | What it does | Use when | | ----------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------------- | | `auto` | Picks `fold`, `dp`, or the existing encoder layout from its width and the request's batch width | Default; you want the decision made per encoder | | `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode | | `dp` | Encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` | | `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP | The two accelerated modes are mutually exclusive per encoder: folding shards the weights for the lifetime of the loaded model, so a folded encoder cannot also be data-parallel. ## Which Mode Wins Measured on H100 across T5 (hidden 4096), Qwen3 (2560), and CLIP-L (768) at batch 1–8 and replica sizes 2 and 4: * **Folding** pays when the encoder is wide enough that sharding its GEMMs beats the per-layer all-reduce it adds. T5 gains; Qwen3 (+35%) and CLIP-L (+50%) get slower, so folding is gated at hidden ≥ 4096. Its benefit also saturates as the replica grows, since each rank's slice keeps shrinking. * **Data-parallel** pays only when the encode is compute-bound, which needs a wide encoder (hidden ≥ 1024 — CLIP-L is slower at every batch and replica measured) and more than one prompt in a single encode call. * **Replication** is the right answer whenever neither condition holds, which is most single-request latency work. `auto` encodes exactly these rules, so prefer it unless you are pinning a configuration you measured yourself. ## Numerics `replicate` matches single-GPU encoding bit-for-bit only when the encoder TP degree is one. Both the existing DiT TP layout and `fold` can reorder parallel reductions; they are mathematically equivalent but are not generally bitwise identical to a single-GPU kernel. `dp` is also not bitwise-identical: each encoder copy sees a smaller batch, so GEMM tiling can differ from the unsplit reference. It may compose with encoder TP: all ranks in one TP group receive the same batch slice, and corresponding TP ranks gather outputs across the orthogonal encoder-DP group. The result remains mathematically equivalent and deterministic for a fixed topology and batch shape, but long video sampling can amplify small floating-point differences. ## Recommended Commands Throughput serving. A single encode call must carry more than one prompt for DP to engage, so raise the batching ceiling too; an encoder flag deliberately does not change DiT batching for you: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --model-type diffusion \ --num-gpus 2 \ --tp-size 1 \ --ulysses-degree 2 \ --encoder-parallel dp \ --batching-max-size 2 ``` Single-request latency with one wide text encoder: ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --model-type diffusion \ --num-gpus 4 \ --ulysses-degree 4 \ --encoder-parallel fold ``` Bit-exact reproducibility against a single-GPU reference: ```bash theme={null} sglang serve \ --model-path Qwen/Qwen-Image-2512 \ --model-type diffusion \ --num-gpus 2 \ --encoder-parallel replicate ``` ## Interaction With Other Flags * **Tensor parallel**: encoder DP composes with TP. A TP group jointly encodes one batch slice; the orthogonal ranks inside the same pipeline replica split and gather the batch. A pure-TP replica has one encoder copy, so there is no additional batch-DP degree. * **Data parallel**: encoder collectives never cross pipeline replicas. With `--dp-size > 1`, each replica independently uses its own TP/SP/CFG ranks. * **Dynamic batching**: `dp` only pays with a wide batch. Selecting it does not change `--batching-max-size`; configure that separately. See [Inference Batching](./dynamic_batching). * **Sequence parallelism**: SP splits the DiT latent sequence. During encoding, those ranks either hold encoder copies for batch DP or join a folded encoder. See [Sequence Parallelism](./ring_sp_performance). # Environment Variables Source: https://docs.sglang.io/docs/sglang-diffusion/environment_variables Configure SGLang diffusion behavior with environment variables. ## Runtime
Environment Variable Default Description
SGLANG\_EXTERNAL\_MODEL\_PACKAGE not set Installed package that registers out-of-tree diffusion pipelines and component models. The package is imported once in every process.
SGLANG\_DIFFUSION\_TARGET\_DEVICE cuda Target device for inference (cuda, rocm, xpu, npu, musa, mps, cpu)
SGLANG\_DIFFUSION\_ATTENTION\_BACKEND not set Override attention backend via env var (e.g. fa, torch\_sdpa, sage\_attn)
SGLANG\_DIFFUSION\_ATTENTION\_CONFIG not set Path to attention backend configuration file (JSON/YAML)
SGLANG\_DIFFUSION\_STAGE\_LOGGING false Enable per-stage timing logs
SGLANG\_DIFFUSION\_SERVER\_DEV\_MODE false Enable dev-only HTTP endpoints for debugging
SGLANG\_DIFFUSION\_TORCH\_PROFILER\_DIR not set Directory for torch profiler traces (absolute path). Enables profiling when set
SGLANG\_DIFFUSION\_CACHE\_ROOT \~/.cache/sgl\_diffusion Root directory for cache files
SGLANG\_DIFFUSION\_CONFIG\_ROOT \~/.config/sgl\_diffusion Root directory for configuration files
SGLANG\_DIFFUSION\_LOGGING\_LEVEL INFO Default logging level
SGLANG\_DIFFUSION\_WORKER\_MULTIPROC\_METHOD fork Multiprocess context for workers (fork or spawn)
SGLANG\_DIFFUSION\_IPC\_A2A true Enable CUDA-IPC all-to-all for eligible same-host, peer-accessible TP1 + two-rank Ulysses groups. Unsupported topology falls back to NCCL; set 0 to force NCCL.
SGLANG\_DIFFUSION\_IPC\_A2A\_TIMEOUT\_MS 10000 Peer-wait timeout in milliseconds. This is a hang backstop, not a per-step budget; raise it only for known long stalls such as layerwise offload.
SGLANG\_DIFFUSION\_IPC\_A2A\_MAX\_BUFFERS 16 Maximum cached IPC staging-buffer shape pairs. Cap this for multi-resolution serving to bound permanent staging memory.
SGLANG\_USE\_RUNAI\_MODEL\_STREAMER true Use Run:AI model streamer for model loading
SGLANG\_KITCHEN\_INT8\_MAX\_ROWS 8192 Max activation rows per kitchen\_int8 fused GEMM. Set 0 to disable row splitting.
SGLANG\_KITCHEN\_INT8\_MIN\_SPLIT\_N 8192 Minimum output features before kitchen\_int8 splits large-M GEMMs. Narrow outputs stay as a single call.
## Platform-Specific ### Apple MPS
Environment Variable Default Description
SGLANG\_USE\_MLX not set SRT only: enables the MLX serving backend. It has no effect on SGLang Diffusion, which uses PyTorch MPS.
### ROCm (AMD GPUs)
Environment Variable Default Description
SGLANG\_USE\_ROCM\_VAE false Use AITer GroupNorm in VAE for improved performance on ROCm
SGLANG\_USE\_ROCM\_CUDNN\_BENCHMARK false Enable MIOpen auto-tuning for VAE conv layers on ROCm
### Quantization
Environment Variable Default Description
SGLANG\_DIFFUSION\_FLASHINFER\_FP4\_GEMM\_BACKEND not set Optional FlashInfer FP4 GEMM backend override for diffusion NVFP4. When unset, SGLang defaults to flashinfer\_trtllm.
SGLANG\_DIFFUSION\_ENABLE\_W8A8\_FP8\_GEMM false Experimental opt-in for fused W8A8 FP8 GEMM in diffusion weight-only FP8 linears. When disabled, FP8 weights are dequantized to the compute dtype before matmul. Enabling this dynamically quantizes activations to FP8 and may change output quality.
## Caching Acceleration These variables configure caching acceleration for Diffusion Transformer (DiT) models. SGLang supports multiple caching strategies - see [caching documentation](./caching-acceleration) for an overview. ### Cache-DiT Configuration See [cache-dit documentation](./cache_dit) for detailed configuration.
Environment Variable Default Description
`SGLANG_CACHE_DIT_ENABLED` false Enable Cache-DiT acceleration
`SGLANG_CACHE_DIT_FN` 1 First N blocks to always compute
`SGLANG_CACHE_DIT_BN` 0 Last N blocks to always compute
`SGLANG_CACHE_DIT_WARMUP` 4 Warmup steps before caching
`SGLANG_CACHE_DIT_RDT` 0.24 Residual difference threshold
`SGLANG_CACHE_DIT_MC` 3 Max continuous cached steps
`SGLANG_CACHE_DIT_TAYLORSEER` false Enable TaylorSeer calibrator
`SGLANG_CACHE_DIT_TS_ORDER` 1 TaylorSeer order (1 or 2)
`SGLANG_CACHE_DIT_SCM_PRESET` none SCM preset (none/slow/medium/fast/ultra)
`SGLANG_CACHE_DIT_SCM_POLICY` dynamic SCM caching policy
`SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` not set Custom SCM compute bins
`SGLANG_CACHE_DIT_SCM_CACHE_BINS` not set Custom SCM cache bins
### Cache-DiT Secondary Transformer For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these variables configure caching for the secondary transformer. Each falls back to its primary counterpart if not set.
Environment Variable Default Description
SGLANG\_CACHE\_DIT\_SECONDARY\_FN (from primary) First N blocks to always compute
SGLANG\_CACHE\_DIT\_SECONDARY\_BN (from primary) Last N blocks to always compute
SGLANG\_CACHE\_DIT\_SECONDARY\_WARMUP (from primary) Warmup steps before caching
SGLANG\_CACHE\_DIT\_SECONDARY\_RDT (from primary) Residual difference threshold
SGLANG\_CACHE\_DIT\_SECONDARY\_MC (from primary) Max continuous cached steps
SGLANG\_CACHE\_DIT\_SECONDARY\_TAYLORSEER (from primary) Enable TaylorSeer calibrator
SGLANG\_CACHE\_DIT\_SECONDARY\_TS\_ORDER (from primary) TaylorSeer order (1 or 2)
## Cloud Storage These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.
Environment Variable Default Description
`SGLANG_CLOUD_STORAGE_TYPE` not set Set to `s3` to enable cloud storage
`SGLANG_S3_BUCKET_NAME` not set The name of the S3 bucket
`SGLANG_S3_ENDPOINT_URL` not set Custom endpoint URL (for MinIO, OSS, etc.)
`SGLANG_S3_REGION_NAME` us-east-1 AWS region name
`SGLANG_S3_ACCESS_KEY_ID` not set AWS Access Key ID
`SGLANG_S3_SECRET_ACCESS_KEY` not set AWS Secret Access Key
## CUDA Crash Debugging These variables enable kernel API logging and optional input/output dumps around diffusion CUDA kernel call boundaries. They are useful when tracking down CUDA crashes such as illegal memory access, device-side assert, or shape mismatches in custom kernels.
Environment Variable Default Description
SGLANG\_KERNEL\_API\_LOGLEVEL 0 Controls crash-debug kernel API logging. 1 logs API names, 3 logs tensor metadata, 5 adds tensor statistics, and 10 also writes dump snapshots.
SGLANG\_KERNEL\_API\_LOGDEST stdout Destination for crash-debug kernel API logs. Use stdout, stderr, or a file path. %i is replaced with the process PID.
SGLANG\_KERNEL\_API\_DUMP\_DIR sglang\_kernel\_api\_dumps Output directory for level-10 kernel API dumps. %i is replaced with the process PID.
SGLANG\_KERNEL\_API\_DUMP\_INCLUDE not set Comma-separated wildcard patterns for kernel API names to include in level-10 dumps.
SGLANG\_KERNEL\_API\_DUMP\_EXCLUDE not set Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps.
# Fused Kernels Source: https://docs.sglang.io/docs/sglang-diffusion/fused_kernels The fused CUDA/Triton kernels SGLang Diffusion ships, what each one replaces, and which are on by default. Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues — each of which is a separate kernel launch and a separate HBM round trip in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion). This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below. ## Two numerical contracts Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. Every kernel in the package falls into one of two classes. **Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch. **Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain. A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR at `quality=high`, which is what motivated the bit-exact rewrite of that path. ## Enabling the request-gated set ```bash theme={null} sglang generate --model-path MODEL_PATH --prompt "..." --quality high ``` The server default stays `lossless`; the OpenAI-compatible endpoints carry it per request. Images: ```bash theme={null} curl -X POST http://${HOST}:${PORT}/v1/images/generations \ -H 'Content-Type: application/json' \ -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}' ``` Video, same field: ```bash theme={null} curl -X POST http://${HOST}:${PORT}/v1/videos \ -H 'Content-Type: application/json' \ -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}' ``` The `quality` field in a **video response** body is unrelated. It is Sora-compatible response metadata and is always reported as `"standard"`; it does not reflect the sampling quality the request ran with. `quality` participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused. Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`. BCG warmup captures the lossless module branches before a high-quality request mounts its DiT fusions, so replay would bypass the requested kernels. SGLang rejects this combination for models with eligible DiT quality sites. Models whose high-quality path changes only VAE decode remain allowed because BCG captures the DiT only. These fusion families mount under `quality="high"`: | Fusion | What it folds | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip | | LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate | | LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch | | Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass | | HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout | | LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels | | SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 | ## Kernel inventory 34 operators are registered in the kernel registry across 38 implementations (some operators carry several backends). Backends are named by provenance, not device: `JIT` compiles under nvcc *and* hipcc, `TRITON` runs on CUDA and ROCm, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, `AOT` comes from the `sgl_kernel` wheel. ### Normalization | Operator | Backend | Contract | Replaces | | -------------------------------------------- | -------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------- | | `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) | | `scale_residual_norm_scale_shift` | Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) | the above plus the preceding `residual + gate * update` | | `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate | | `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k | | `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm | | `norm_scale_shift` | CuTe-DSL / FlyDSL | fp32 statistics | LN-or-RMS + scale/shift, many broadcast modes | | `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` | | `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous | | `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels\_last end to end with no `nchwToNhwc` transposes | | `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU | ### adaLN modulation and gating | Operator | Backend | Contract | Replaces | | ---------------------- | -------- | --------- | ------------------------------------------------------------ | | `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` | | `residual_gate_add` | JIT CUDA | bit-exact | `residual + gate * update` | | `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding | | `temb_table_slices` | Triton | bit-exact | see note below | | `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous | `temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops. ### RoPE and QK-norm | Operator | Backend | Contract | Replaces | | --------------------------- | -------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE | | `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection | | `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels | | `ltx2_qknorm_split_rope` | JIT CUDA | close (validated on B200) | LTX-2 QK-norm + split RoPE | | `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables | | `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass | ### Activation | Operator | Backend | Contract | Replaces | | ----------------------- | -------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `silu_mul` | Triton | bit-exact | `F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat` | | `bias_silu`, `bias_glu` | Triton | bit-exact | Sana GLUMB conv bias + SiLU / GLU post-processing | | `linear_gelu_tanh` | AOT (cublasLt) | not bit-exact, request-gated | bias-add and tanh-GELU folded into the GEMM epilogue | ### Attention | Operator | Backend | Notes | | ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention | | `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in | ### Data movement Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces. | Operator | Backend | Replaces | | --------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------- | | `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) | | `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack | | `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path | | `causal_conv3d_cat_pad` | JIT CUDA / Triton | causal Conv3d `cat` + `pad` | | `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass | | `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` | ## Coverage by model Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal. | Model | Fused paths | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU | | FLUX.2 | LN+modulate, packed SwiGLU, residual-gate add | | Qwen-Image | linear+GELU, select-0/1 LN modulation | | GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU | | ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add | | Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm | | Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add | | LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU | | LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache | | HunyuanVideo | QKV+RoPE pack, strided QK RMSNorm, linear+GELU | | LingBot Video MoE | Fused RMSNorm at `quality=high` | | Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add | | SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; LN+modulate, GLUMB bias+SiLU / bias+GLU, and residual-gate add during BCG; BF16-input linear attention at `quality=high` | | Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS | | Wan | temb table slices; VAE cat+pad and DupUp3D add, `channels_last_3d` RMSNorm+SiLU | | Cosmos3 / Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) | | FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) | ## Inspecting what is registered Every kernel is described by a `KernelSpec` in the process-wide registry, so the inventory is queryable without importing any backend: ```python theme={null} from sglang.kernels.registry import registry diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")] for spec in registry.get("diffusion.scale_residual_norm_scale_shift"): print(spec.backend, spec.target, spec.capabilities) ``` Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several: ```python theme={null} from sglang.kernels import select_kernel, KernelBackend fn = select_kernel( "diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL ).load() ``` ## Importing the kernels Runtime code imports from the package, never from a submodule: ```python theme={null} from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact ``` Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call `can_use_(...)` first and fall back to the reference chain when it returns `False`; the kernel raises on an unsupported input rather than silently returning `None`. The package `README.md` carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed. ## References * [Performance Optimization](./performance-optimization) * [Attention Backends](./attention_backends) * [Quantization](./quantization) * [Profiling](./profiling) * [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix * [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace # SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/index Accelerated image and video generation with diffusion models. SGLang Diffusion is a high-performance inference framework for image and video generation. It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths. ## Key Features * Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more * Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration * Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API * Multi-platform support for NVIDIA, AMD, Intel XPU, Ascend, Apple Silicon, and Moore Threads ## Quick Start ```bash theme={null} uv pip install "sglang[diffusion]" --prerelease=allow ``` ```bash theme={null} sglang generate --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` ```bash theme={null} sglang serve --model-path Qwen/Qwen-Image --port 30010 ``` ## Start Here * [Installation](/docs/sglang-diffusion/installation): install SGLang Diffusion and platform dependencies * [Supported Models and Optimization Compatibility](/docs/sglang-diffusion/compatibility_matrix): check supported model families, long-tail coverage, and optimization support * [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server * [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server * [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers * [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost * [Quantization](/docs/sglang-diffusion/quantization): configure component checkpoint and causal KV-cache quantization * [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models): understand session state, causal caches, and realtime-only controls * [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines ## Additional Documentation * [Post-Processing](/docs/sglang-diffusion/api/post_processing): frame interpolation and upscaling * [Models with AR Stage](/docs/sglang-diffusion/models_with_ar): run hybrid diffusion pipelines like GLM-Image with a separate AR encoder server * [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe): run ERNIE-Image with either built-in PE or a separate PE server * [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook): choose `--performance-mode`, offload, FSDP, CFG parallelism, SP, and TP * [Attention Backends](/docs/sglang-diffusion/attention_backends): choose the best backend for your model and hardware * [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance): configure SP, Ulysses, and ring-based splitting for long sequences * [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel): fold, data-parallel, or replicate the text/image encoders across idle GPUs * [Inference Batching](/docs/sglang-diffusion/dynamic_batching): batch compatible native diffusion requests during serving * [Progressive Resolution Generation](/docs/sglang-diffusion/progressive_resolution): run early denoising steps at lower latent resolution for selected pipelines * [Environment Variables](/docs/sglang-diffusion/environment_variables): platform, caching, storage, and debugging configuration ## Developer Documentation * [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for new diffusion pipelines * [CI Performance Baselines](/docs/sglang-diffusion/ci_perf): generate and update performance baselines used in CI ## References * [SGLang GitHub](https://github.com/sgl-project/sglang) * [Cache-DiT](https://github.com/vipshop/cache-dit) * [FastVideo](https://github.com/hao-ai-lab/FastVideo) * [xDiT](https://github.com/xdit-project/xDiT) * [Diffusers](https://github.com/huggingface/diffusers) # Install SGLang Diffusion Source: https://docs.sglang.io/docs/sglang-diffusion/installation Install SGLang Diffusion on NVIDIA, AMD, MUSA, and Ascend platforms. You can install SGLang-Diffusion using one of the methods below. The standard installation already includes SGLang's optimized kernel stack, including both `sgl-kernel` and JIT kernels used by diffusion workloads. ## Standard Installation (NVIDIA GPUs) ### Method 1: With pip or uv It is recommended to use uv for a faster installation: ```bash Command theme={null} pip install --upgrade pip pip install uv uv pip install "sglang[diffusion]" --prerelease=allow ``` ### Method 2: From source ```bash Command theme={null} # Use the latest release branch git clone https://github.com/sgl-project/sglang.git cd sglang # Install the Python packages pip install --upgrade pip pip install -e "python[diffusion]" # With uv uv pip install -e "python[diffusion]" --prerelease=allow ``` ### Method 3: Using Docker The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang), built from the [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile). Replace `` below with your HuggingFace Hub [token](https://huggingface.co/docs/hub/en/security-tokens). ```bash Command theme={null} docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:dev \ zsh -c '\ echo "Installing diffusion dependencies..." && \ pip install -e "python[diffusion]" && \ echo "Starting SGLang-Diffusion..." && \ sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --prompt "A logo With Bold Large text: SGL Diffusion" \ --save-output \ ' ``` ## Platform-Specific: ROCm (AMD GPUs) For AMD Instinct GPUs (e.g., MI300X), use a ROCm-enabled Docker image from [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang). The tag below is an example for ROCm 7.0 on MI300X and may lag the latest release tag: ```bash Command theme={null} docker run --device=/dev/kfd --device=/dev/dri --ipc=host \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env HF_TOKEN= \ lmsysorg/sglang:v0.5.5.post2-rocm700-mi30x \ sglang generate --model-path black-forest-labs/FLUX.1-dev --prompt "A logo With Bold Large text: SGL Diffusion" --save-output ``` For detailed ROCm system configuration and installation from source, see [AMD GPUs](../hardware-platforms/amd_gpu). ## Platform-Specific: MUSA (Moore Threads GPUs) For Moore Threads GPUs (MTGPU) with the MUSA software stack, follow the platform guide first. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching: ```bash Command theme={null} # Clone the repository git clone https://github.com/sgl-project/sglang.git cd sglang # Install the Python packages pip install --upgrade pip mv python/pyproject.toml python/pyproject.toml.bak cp python/pyproject_other.toml python/pyproject.toml pip install -e "python[all_musa]" ``` ## Platform-Specific: Intel XPU For Intel Data Center GPU Max or Arc GPUs, follow the installation with docker in [XPU installation guide](../hardware-platforms/xpu). The dockerfile already include diffusion dependencies: ## Platform-Specific: Ascend NPU For Ascend NPU, please follow the [NPU installation guide](../hardware-platforms/ascend-npus/getting-started/installation). Quick test: ```bash Command theme={null} sglang generate --model-path black-forest-labs/FLUX.1-dev \ --prompt "A logo With Bold Large text: SGL Diffusion" \ --save-output ``` ## Platform-Specific: Apple MPS For Apple MPS, follow the instructions below to install from source. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching: ```bash Command theme={null} # Install ffmpeg brew install ffmpeg # Install uv brew install uv # Clone the repository git clone https://github.com/sgl-project/sglang.git cd sglang # Create and activate a virtual environment uv venv -p 3.12 sglang-diffusion source sglang-diffusion/bin/activate # Install the Python packages uv pip install --upgrade pip mv python/pyproject.toml python/pyproject.toml.bak cp python/pyproject_other.toml python/pyproject.toml uv pip install -e "python[all_mps]" ``` SGLang Diffusion uses PyTorch MPS. The `all_mps` extra also installs the SRT MLX backend dependencies; `SGLANG_USE_MLX` applies only to SRT serving. # Diffusion models with autoregressive stages Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_ar Run diffusion pipelines with in-process or separately deployed autoregressive encoders. SGLang Diffusion supports two AR execution paths. Qwen Image Layered and LongCat-Image use the native Qwen2.5-VL component in process. GLM-Image can use its bundled Transformers implementation or delegate AR inference to a separate SGLang server. ## GLM-Image Quick Start Run GLM-Image with its bundled Transformers implementation (default): ```bash theme={null} # Terminal 1 : launch server sglang serve --model-path zai-org/GLM-Image --port ${PORT} ``` ```bash theme={null} # Terminal 2 : launch client curl http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "prompt", "n": 1, "size": "widthxheight" }' ``` Run GLM-Image with a separate SGLang server for its AR stage: ```bash theme={null} # Terminal 1 : launch server with AR model sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal --port ${AR_PORT} ``` ```bash theme={null} # Terminal 2 : launch server with Diffusion model sglang serve --model-path /path/to/zai-org/GLM-Image/ \ --srt-encoder-url "http://${HOST}:${AR_PORT}" \ --port ${PORT} ``` ```bash theme={null} # Terminal 3 : launch client curl http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "prompt", "n": 1, "size": "widthxheight" }' ``` ## Support matrix
Model Transformers backend SGLang backend
GLM-Image T2I, I2I, V2I T2I
Qwen Image Layered Not used I2I, in-process native Qwen2.5-VL
LongCat-Image Not used T2I, in-process native Qwen2.5-VL
## Deployment Assumptions & Limitations :::warning **Network Latency & Timeouts:** In SGLang backend mode, the Diffusion server sends an HTTP request to `--srt-encoder-url` for **every auto-regressive (AR) step**. * To prevent requests from breaking during long model generations, increase `--srt-encoder-timeout` (e.g., set to 100 seconds). * To protect the system against temporary network delays or brief drops in connection, use `--srt-encoder-connection-timeout`. ::: * **Recommended Setup:** Run both servers on the same machine or inside the same fast local network. * **Cross-Region Warning:** Running the Diffusion server and the AR server in different geographic regions will slow down token generation and heavily reduce performance. * **Startup Connection Check:** SGLang automatically checks the connection to `--srt-encoder-url` when starting up. The server will stop immediately if the remote AR host is offline. ## Ascend NPU ENV To run 2 servers on same group of NPU you need to specify env variables [https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref\_07\_0144.html](https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref_07_0144.html) Example: ```bash theme={null} # Terminal 1 : server with AR model export HCCL_IF_BASE_PORT=23000 export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199" export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399" ``` ```bash theme={null} # Terminal 2 : server with diffusion model export HCCL_IF_BASE_PORT=24000 export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199" export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399" ``` ## Best practices GLM-Image example for Ascend A3 2 cards (4 devices) ```bash theme={null} # Terminal 1 : server with AR model export HCCL_IF_BASE_PORT=23000 export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199" export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399" sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \ --cuda-graph-bs 1 --device npu --attention-backend ascend --image-processor-backend pil \ --tp-size 4 --port ${PORT} --mem-fraction-static 0.4 ``` Second terminal with diffusion server: ```bash theme={null} # Terminal 2 : run SGL-Diffusion generate command export HCCL_IF_BASE_PORT=24000 export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199" export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399" SGLANG_CACHE_DIT_FN=2 SGLANG_CACHE_DIT_BN=1 SGLANG_CACHE_DIT_WARMUP=4 SGLANG_CACHE_DIT_RDT=0.4 \ SGLANG_CACHE_DIT_MC=4 SGLANG_CACHE_DIT_TAYLORSEER=true SGLANG_CACHE_DIT_TS_ORDER=2 \ SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path /path/to/zai-org/GLM-Image/ \ --prompt "A curious raccoon" --height 1920 --width 1088 --num-inference-steps 50 --num-gpus 4 \ --sp-degree 4 --srt-encoder-url "http://${HOST}:${PORT}" --warmup-mode request ``` Result: ```bash theme={null} Warmed-up request processed in 33.82 seconds (with warmup excluded) ``` # Diffusion Models with Prompt Enhancement (PE) Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_pe Run ERNIE-Image with built-in prompt enhancement or a separate SGLang-served PE model. ## Quick Start By default, the diffusion server loads the PE model in-process with SGLang's native Ministral3 implementation. Deploy the PE model as a separate SGLang server when it needs independent resources or scaling. This document uses `baidu/ERNIE-Image` as an example. Run the model with the built-in native PE implementation (default): ```bash theme={null} # Terminal 1: launch server sglang serve --model-path baidu/ERNIE-Image --port ${PORT} ``` ```bash theme={null} # Terminal 2: launch client curl -X POST http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "This is a photograph depicting an urban street scene. Shot at eye level, it shows a covered pedestrian or commercial street. Slightly below the center of the frame, a cyclist rides away from the camera toward the background, appearing as a dark silhouette against backlighting with indistinct details. The ground is paved with regular square tiles, bisected by a prominent tactile paving strip running through the scene, whose raised textures are clearly visible under the light. Light streams in diagonally from the right side of the frame, creating a strong backlight effect with a distinct Tyndall effect—visible light beams illuminating dust or vapor in the air and casting long shadows across the street. Several pedestrians appear on the left side and in the distance, some with their backs to the camera and others walking sideways, all rendered as silhouettes or semi-silhouettes. The overall color palette is warm, dominated by golden yellows and dark browns, evoking the atmosphere of dusk or early morning.", "height": 1264, "width": 848, "num_inference_steps": 50, "guidance_scale": 4.0, "use_pe": true }' ``` Run the model with an SGLang-served PE model (high performance): ```bash theme={null} # Terminal 1: launch SGLang PE model server sglang serve --model-path /path/to/baidu/ERNIE-Image/pe/ --port ${PE_PORT} ``` ```bash theme={null} # Terminal 2: launch diffusion model server with PE server sglang serve --model-path /path/to/baidu/ERNIE-Image/ \ --pe-server-url "http://${HOST}:${PE_PORT}" \ --port ${PORT} ``` ```bash theme={null} # Terminal 3: launch client curl -X POST http://${HOST}:${PORT}/v1/images/generations \ -H "Content-Type: application/json" \ -d '{ "prompt": "This is a photograph depicting an urban street scene. Shot at eye level, it shows a covered pedestrian or commercial street. Slightly below the center of the frame, a cyclist rides away from the camera toward the background, appearing as a dark silhouette against backlighting with indistinct details. The ground is paved with regular square tiles, bisected by a prominent tactile paving strip running through the scene, whose raised textures are clearly visible under the light. Light streams in diagonally from the right side of the frame, creating a strong backlight effect with a distinct Tyndall effect—visible light beams illuminating dust or vapor in the air and casting long shadows across the street. Several pedestrians appear on the left side and in the distance, some with their backs to the camera and others walking sideways, all rendered as silhouettes or semi-silhouettes. The overall color palette is warm, dominated by golden yellows and dark browns, evoking the atmosphere of dusk or early morning.", "height": 1264, "width": 848, "num_inference_steps": 50, "guidance_scale": 4.0, "use_pe": true }' ``` For a memory-constrained in-process deployment, the native PE decoder supports layerwise offload: ```bash theme={null} sglang serve --model-path baidu/ERNIE-Image \ --layerwise-offload-components pe \ --port ${PORT} ``` This option streams PE decoder layers from CPU and can increase prompt-enhancement latency. It does not apply when `--pe-server-url` selects an external PE server. ## Support matrix | Model | Built-in PE | SGLang PE Server | | ----------- | ----------- | ---------------- | | ERNIE-Image | ✅ | ✅ | ## Ascend NPU Environment See [Diffusion models with AR stage like GLM-Image](/docs/sglang-diffusion/models_with_ar#ascend-npu-env). # Parallelism Overview Source: https://docs.sglang.io/docs/sglang-diffusion/parallelism SGLang Diffusion ships several parallelism strategies. Each one splits a different dimension of the DiT forward pass, which is exactly why they can be combined: the total GPU count is the product of the degrees, ```text theme={null} num_gpus = cfg_parallel_degree × tp_size × sp_degree sp_degree = ulysses_degree × ring_degree ``` This page is the map — what each axis does, which combinations are legal, and how to pick one. Per-axis depth lives in [Sequence Parallelism](./ring_sp_performance), [Encoder Parallelism](./encoder_parallel) (text/image encoders are a separate axis with their own knob), and the [CLI reference](./api/cli). ## The axes | Strategy | Splits | Communication | Flag | | -------------------- | -------------------------------------------- | --------------------------------------------------- | --------------------- | | CFG parallel | guidance branches | one combine per denoise step | `--cfg-parallel-size` | | Tensor parallel (TP) | weights and attention heads | all-reduce per transformer block | `--tp-size` | | Ulysses SP | sequence outside attention ↔ heads inside it | two all-to-alls per attention | `--ulysses-degree` | | Ring SP | sequence rows inside attention | neighbor-only K/V rotation, overlapped with compute | `--ring-degree` | | K/V-gather CP | sequence rows inside attention | one K/V all-gather per attention | `--kv-gather-degree` | | Data parallel | requests | none between replicas | `--dp-size` | Two strategies compose when they split different dimensions. TP and Ulysses both touch heads but compose serially — TP splits the projection weights, then Ulysses splits the activations of the TP-local heads. Ring and Ulysses compose because ring splits rows while Ulysses splits heads. Ring has no composition with a K/V-all-gather style of attention parallelism: both answer the same question (how a rank's query rows see remote K/V), so they are alternatives for one slot, not complements. ## What happens to the shapes For `tp_size = T`, `ulysses_degree = U`, `ring_degree = R`, one attention runs: ```text theme={null} [B, S/(U·R), H/T, D] sequence-sharded activations │ Ulysses input all-to-all (inside each Ulysses group) ▼ [B, S/R, H/(T·U), D] full sequence of this ring block, few heads │ ring attention (R−1 neighbor hops; Q never moves, K/V rotate) ▼ [B, S/R, H/(T·U), D] │ Ulysses output all-to-all (inverse) ▼ [B, S/(U·R), H/T, D] ``` The ring merge (online softmax) requires every rank in a ring group to hold the *same heads* over *different rows*; the group construction guarantees this. A K/V-gather (CP-style) variant fills the same slot differently: instead of R−1 overlapped hops it all-gathers K/V once and computes the local Q rows against the full sequence in one shot — fewer, larger transfers, paid for by holding the whole K/V per rank. Like ring it splits rows, so it adds no head constraint. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` — its measured-win zone — and higher degrees default to Ulysses. Ulysses groups are laid out on contiguous ranks and ring groups on strided ranks, so with a node-major rank mapping, Ulysses traffic stays on intra-node NVLink (all-to-all needs full-bisection bandwidth) while ring hops cross the slower interconnect where neighbor-only transfers overlap with compute. A mis-mapped layout stays numerically correct and silently loses the performance — worth checking when a sharded run is unexpectedly slow. ## Constraints * `num_attention_heads % tp_size == 0` — TP splits heads at the projections. * `(num_attention_heads / tp_size) % ulysses_degree == 0` — Ulysses splits the **TP-local** heads. `H % U == 0` alone is not sufficient: 56 heads pass with `tp=2, ulysses=4` (28 % 4) and fail with `tp=4, ulysses=4` (14 % 4). * Ring adds no head constraint (it splits rows), but the sequence — including any model-specific packing alignment — must divide by `ulysses × ring`, since ring adds an outer row split on top of Ulysses's inner one. * Ring requires an attention backend that declares `supports_ring_rotation()` — the per-hop merge needs the kernel's softmax LSE. `fa` and `sage_attn` declare it; the launcher auto-selects `fa` when unset. * `USPAttention`'s masked/tail-padded text path and its replicated-prefix, -suffix, and -kv-prefix paths all support ring: the sharded K/V rotates through the ring while the tail-pad or replicated portion is attended locally once and combined into the ring result with the same online-softmax merge. This covers the joint text+image attention most models use (flux, flux\_2, qwen\_image, zimage, glm\_image, ernie\_image, and others). * What still raises `NotImplementedError` under ring: `USPAttention`'s generic varlen path (multiple packed segments per row, as used by HunyuanVideo — no ring-aware rotation for it yet), and the legacy stacked-QKV `UlyssesAttention` layer, now down to one user (Wan's VSA sparse-attention variant) that has no softmax LSE to merge and can't gain ring support without a different kernel. * The launcher validates `num_gpus` against the product of the degrees and fails fast on any mismatch. ## The Ulysses transport The all-to-alls normally run over NCCL. On exactly 2 GPUs with peer-to-peer access, a CUDA-IPC transport replaces them by default: each rank writes its half directly into the peer's mapped staging buffer, with GPU-side sequence counters instead of a NCCL rendezvous. An all-to-all is a permutation, never a reduction, so the transport cannot change results — outputs are bitwise identical to the NCCL path. | Environment variable | Default | Meaning | | -------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `SGLANG_DIFFUSION_IPC_A2A` | `1` | set `0` to force NCCL | | `SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS` | `10000` | deadlock backstop for the peer wait; on expiry the transport retires on every rank and the request fails rather than returning incomplete data | | `SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS` | `16` | staging pairs kept alive; raise for many-resolution serving | Independently of the transport, the default path already packs the three Q/K/V input exchanges into one destination-major collective. A handful of models (e.g. LTX-2) instead opt into `enable_packed_qkv_input_a2a`, which pipelines three separate exchanges over a dedicated stream rather than merging them into one payload — a different trade-off, not a strict upgrade over the default. ## Which axes tolerate crossing nodes Each axis has a fixed communication pattern, and the pattern — volume per step, how often it fires, and whether it can hide behind compute — decides whether the axis survives the drop from NVLink to the inter-node fabric. Ordered from most to least cross-node friendly: | Axis | Pattern | Traffic per denoise step | Cross-node verdict | | --------------- | -------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Data parallel | none between replicas | zero on the request path | **Best.** Replicas only share startup init and control-op fan-out; no fast interconnect needed at all. | | CFG parallel | one branch-combine | one latent-sized exchange, once | **Good candidate.** Once per step, small payload, naturally deadline-tolerant. Unmeasured across nodes so far. | | K/V-gather CP | one K/V all-gather per attention | (R−1)/R of K/V, one unoverlapped burst | **Between Ulysses and ring:** only K/V moves (queries never do), but the burst cannot hide behind compute — prefer ring across nodes, gather at small degrees within one. | | Ring SP | neighbor-only K/V rotation | K/V ÷ ring\_degree, (R−1) hops, overlapped with attention tiles | **Designed for it.** The mechanism now covers most models' joint attention (see Constraints). Actually crossing nodes end-to-end is validated for MiniMax H3 (Ulysses intra-node × ring across, net positive and growing with sequence length); other models' ring support is same-node-validated so far. | | Ulysses SP | all-to-all | full q/k/v activations, twice per attention, every layer | **Keep intra-node.** All-to-all needs full-bisection bandwidth; across nodes it becomes R² flows with receiver incast. | | Tensor parallel | all-reduce | hidden-sized reduction per transformer block (×60 blocks for qwen-class DiTs) | **Worst.** Highest frequency, no overlap, already \~70% of sharded kernel time on NVLink. | Two caveats keep this a map rather than a promise. Cross-node launch (`--nnodes`/`--node-rank`/`--dist-init-addr`) is merged, and per-model ring support is broad now (see the constraints above) — but those two facts together still don't add up to "any model, any node count." The only configuration actually run end-to-end across nodes is the H3 recipe; other models' ring support is same-node-validated so far, which means crossing nodes with them is untested, not disallowed. And the data-parallel row describes the design: the current implementation binds each replica's ingress on the local host, so replicas spanning hosts additionally need per-replica host addressing before `--dp-size` can place one replica per node. ## Choosing a configuration Measured guidance rather than rules — the right combination depends on the model's communication profile and the hardware topology, and legal does not mean profitable: * **Multi-branch (true-CFG) models**: CFG parallelism first. Branches run the whole DiT independently and combine once per step, avoiding per-layer communication entirely. * **Single-branch image models on 2 GPUs**: Ulysses and TP trade places by model. Communication-heavy DiTs measured faster with Ulysses; smaller DiTs with TP. Measure both; do not copy a winner across models. * **Long video / packed sequences**: Ulysses up to the head-divisibility limit, then ring for the remaining factor — sequence length scales past the head count where Ulysses alone cannot. * **When Ulysses's head divisibility blocks the degree you need** (`H/T` not divisible by the target `U`): the row-splitting slot sidesteps it — ring today, or `--kv-gather-degree` (it splits rows, so it adds no head constraint either). * **TP beyond 2 ranks** rarely improves image-DiT latency: the per-block all-reduce grows with rank count faster than the GEMM savings. ## Data parallelism `--dp-size N` runs N full engine replicas on `num_gpus / N` GPUs each, every replica with its own ingress. Generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica; replicas exchange nothing on the request path. Monolithic serving only, and the ingress currently binds on the local host — one replica per node needs per-replica host addressing first. # Performance Optimization Source: https://docs.sglang.io/docs/sglang-diffusion/performance-optimization Choose performance levers for SGLang Diffusion by latency, throughput, memory, and quality tradeoffs. Use this page as the starting point for SGLang Diffusion performance work. It separates performance levers into two decision classes: * **Output-preserving / lossless-style:** system settings that should preserve model behavior while changing residency, parallelism, kernels, or scheduling. * **Quality-tradeoff / lossy or approximate:** techniques that can change the denoising path, numerical representation, or generated output. The docs use "output-preserving" instead of promising bit-exact "lossless" because different kernels, GPU types, or precision paths can still introduce small numerical differences. The decision boundary is whether the optimization intentionally trades quality or output equivalence for speed. ## Start Here 1. Pick a serving or generation mode from [Deployment and Performance Modes](./deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit. 2. Choose the right attention backend from [Attention Backends](./attention_backends). 3. Use [Sequence Parallelism](./ring_sp_performance) only when the model and video shape benefit from sequence splitting. 4. Use [Inference Batching](./dynamic_batching) for concurrent compatible requests during serving. 5. Use [Profiling](./profiling) before changing several levers at once. ## Output-Preserving / Lossless-Style Levers These settings should preserve model behavior while changing residency, parallelism, kernels, or scheduling. They are the first choices for production tuning.
Lever Use when Docs
--performance-mode You want a safe preset for speed or memory without overriding explicit flags. Deployment and Performance Modes
Offload, FSDP, CFG parallelism GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck. Deployment and Performance Modes
Sequence parallelism Long image/video sequences need sequence-level parallelism. Sequence Parallelism
--encoder-parallel Text/image encoding is a visible share of the request and the DiT replica sits idle during it. Encoder Parallelism
Attention backend Kernel choice dominates DiT latency or memory. Attention Backends
Fused kernels You want to know which elementwise chains are already fused, or to opt into the request-gated set. Fused Kernels
Dynamic batching Serving many compatible requests concurrently. Inference Batching
## Quality-Tradeoff / Lossy Or Approximate Levers These techniques can change the denoising path, numerical representation, or generated output. They are useful after you have a baseline and an acceptance criterion for quality.
Lever Tradeoff Docs
Cache-DiT Skips selected DiT block or step computation based on cache decisions. Cache-DiT
TeaCache Reuses residuals when consecutive denoising steps are similar enough. TeaCache
Progressive resolution Runs early denoising at lower latent resolution for supported pipelines. Progressive Resolution Generation
Quantization Uses lower-precision transformer weights or activations. Quantization
## Practical Order 1. Establish a baseline with the target model, resolution, frame count, step count, and GPU type. 2. Select `--performance-mode` and explicit residency or parallelism flags. 3. Tune attention backend and batching for the deployment pattern. 4. Profile if the bottleneck is unclear. 5. Add caching, progressive resolution, or quantization only after comparing output quality against your acceptance target. ## Diagnostics [Profiling](./profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers. ## References * [Deployment and Performance Modes](./deployment_cookbook) * [Attention Backends](./attention_backends) * [Fused Kernels](./fused_kernels) * [Sequence Parallelism](./ring_sp_performance) * [Caching Strategies](./caching-acceleration) * [Profiling](./profiling) # Profiling Source: https://docs.sglang.io/docs/sglang-diffusion/profiling Profile SGLang diffusion workloads with PyTorch Profiler and Nsight Systems. This guide covers profiling techniques for multimodal generation pipelines in SGLang. ## PyTorch Profiler PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics. ### Denoising Stage Profiling Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step): ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 \ --profile ``` **Parameters:** * `--profile`: Enable profiling for the denoising stage * `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5) * Smaller values reduce trace file size * Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step ### Full Pipeline Profiling Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.): ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 \ --profile \ --profile-all-stages ``` **Parameters:** * `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising ### Output Location By default, trace files are saved in the ./logs/ directory. The exact output file path will be shown in the console output, for example: ```bash theme={null} [mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz ``` ### View Traces Load and visualize trace files at: * [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (recommended) * chrome://tracing (Chrome only) For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`. ### `--perf-dump-path` (Stage/Step Timing Dump) Besides profiler traces, you can also dump a lightweight JSON report that contains: * stage-level timing breakdown for the full pipeline * step-level timing breakdown for the denoising stage (per diffusion step) This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike). The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key. Example: ```bash theme={null} sglang generate \ --model-path \ --prompt "" \ --perf-dump-path perf.json ``` ## Nsight Systems Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns. ### Installation See the [SGLang profiling guide](../developer_guide/benchmark_and_profiling#profile-with-nsight) for installation instructions. ### Basic Profiling Profile the entire pipeline execution: ```bash theme={null} nsys profile \ --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --force-overwrite=true \ -o QwenImage \ sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 ``` ### Targeted Stage Profiling Use `--delay` and `--duration` to capture specific stages and reduce file size: ```bash theme={null} nsys profile \ --trace-fork-before-exec=true \ --cuda-graph-trace=node \ --force-overwrite=true \ --delay 10 \ --duration 30 \ -o QwenImage_denoising \ sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --seed 0 ``` **Parameters:** * `--delay N`: Wait N seconds before starting capture (skip initialization overhead) * `--duration N`: Capture for N seconds (focus on specific stages) * `--force-overwrite`: Overwrite existing output files ## Notes * **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems * **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline * **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads ## FAQ * If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time. # Progressive Resolution Generation Source: https://docs.sglang.io/docs/sglang-diffusion/progressive_resolution Experimental spectral progressive resolution growing for selected SGLang Diffusion pipelines. Progressive resolution growing is an experimental feature for selected SGLang Diffusion pipelines. It runs early denoising steps at a coarser latent resolution and spectrally upsamples the latent before the full-resolution steps. On the benchmark setup below, this reduces the quadratic attention cost of the DiT transformer and yields up to **1.63× speedup on FLUX.1**, **1.93× speedup on FLUX.2**, **2.33× speedup on Z-Image**, **2.78× speedup on Wan 2.1 T2V**, **1.69× speedup on Qwen-Image**, and **1.56× speedup on Ideogram 4**. Based on [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736). ## Overview DiT attention is O(n²) in sequence length. Running the first N denoising steps at half the spatial resolution cuts the attention cost to \~6% for those steps. The transition point — how many steps to run at each resolution — is computed from the **Bayes-optimal frequency-activation criterion**: frequencies that cannot be resolved at the coarse scale are not denoised there. The method is designed to preserve quality under this criterion, but generated outputs can still differ from the full-resolution baseline. | Model | Full-res tokens | Half-res tokens | Token-step ratio | | ------------------------------- | --------------- | --------------- | ---------------- | | FLUX.1 1024×1024 | 4,096 | 1,024 | 4.0× | | FLUX.2 1024×1024 | 4,096 | 1,024 | 4.0× | | Z-Image 1024×1024 | 4,096 | 1,024 | 4.0× | | Wan 2.1 T2V 480×832 (81 frames) | 6,240 | 1,560 | 4.0× | | Ideogram 4 1024×1024 | 4,096 | 1,024 | 4.0× | ## Parameters | Parameter | CLI flag | Default | Description | | -------------------- | ---------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `progressive_mode` | `--progressive-mode` | `"fullres"` | `"fullres"` disables (identical to standard generation). `"dct_rewind"` enables spectral upsample with scheduler rewind (recommended). `"dct"` enables upsample without rewind. | | `progressive_levels` | `--progressive-levels` | `1` | Number of resolution halvings. `1` = one coarse stage (64×64 latent → 128×128). `2` = two coarse stages (32×32 → 64×64 → 128×128). | | `progressive_delta` | `--progressive-delta` | `0.01` | Noise-dominated tolerance δ. Controls how many steps run at coarse resolution. Higher δ = more coarse steps = more speedup. | > **Tip:** Add `--dit-cpu-offload false` to keep the transformer GPU-resident. With CPU offload each step pays a fixed PCIe transfer cost regardless of sequence length, which dilutes the speedup. *** ## FLUX.1 ### Usage ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --num-inference-steps 50 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | ------ | ----------------------- | ----------------- | | `0.01` | 18 @ 64² + 32 @ 128² | **1.32×** | | `0.05` | 28 @ 64² + 22 @ 128² | **1.63×** | For most prompts `0.05` is recommended — it gives the largest speedup with no visible degradation. ### Benchmark Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only. | Config | Stage split | Denoise | Speedup | | --------------------- | --------------------------- | ------- | --------- | | Fullres (baseline) | 50 @ 128² latent | 36.65 s | 1.00× | | dct\_rewind L1 δ=0.01 | 18\@64² + 32\@128² | 27.67 s | **1.32×** | | dct\_rewind L1 δ=0.05 | 28\@64² + 22\@128² | 22.58 s | **1.62×** | | dct\_rewind L2 δ=0.01 | 10\@32² + 8\@64² + 32\@128² | 26.48 s | **1.38×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="black-forest-labs/FLUX.1-dev", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 50, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## FLUX.2 Supports `FLUX.2-dev`, `FLUX.2-klein-4B`, and `FLUX.2-klein-9B`. ### Usage ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-klein-4B \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --num-inference-steps 30 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.10 ``` ### Benchmark Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Model: FLUX.2-klein-4B, 30 steps, 1024×1024. Timing = denoising loop only, averaged across 10 diverse prompts. | Config | Stage split | Denoise | Speedup | | --------------------- | ----------------- | ------- | --------- | | Fullres (baseline) | 30 @ 64² latent | 9.72 s | 1.00× | | dct\_rewind L1 δ=0.05 | 18\@32² + 12\@64² | 5.50 s | **1.77×** | | dct\_rewind L1 δ=0.10 | 20\@32² + 10\@64² | 5.03 s | **1.93×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="black-forest-labs/FLUX.2-klein-4B", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 30, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.10, }) ``` *** ## Wan 2.1 T2V Supports `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` and `Wan-AI/Wan2.1-T2V-14B-Diffusers`. > **Note:** Progressive generation grows only the **spatial** H×W dimensions. The temporal dimension T (number of latent frames) is kept fixed across all stages. ### Usage ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic" \ --num-inference-steps 50 \ --num-frames 81 \ --height 480 \ --width 832 \ --guidance-scale 5.0 \ --flow-shift 5.0 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | ------ | ------------------------ | ----------------- | | `0.01` | 23 @ 30×52 + 27 @ 60×104 | **1.65×** | | `0.02` | 27 @ 30×52 + 23 @ 60×104 | **1.86×** | | `0.05` | 33 @ 30×52 + 17 @ 60×104 | **2.32×** | | `0.10` | 37 @ 30×52 + 13 @ 60×104 | **2.78×** | For most prompts `0.05` is recommended. `0.10` provides maximum speedup but should be validated on motion-heavy scenes. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="Wan-AI/Wan2.1-T2V-1.3B-Diffusers", dit_cpu_offload=False, flow_shift=5.0, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic", "num_inference_steps": 50, "num_frames": 81, "height": 480, "width": 832, "guidance_scale": 5.0, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## Z-Image Supports `Tongyi-MAI/Z-Image`. Z-Image uses the same VAE as FLUX.1 (`FluxVAEConfig`), so the power-law spectrum constants are identical. The progressive stage handles Z-Image's 5-D latent format `[B, C, 1, H, W]` with squeeze/unsqueeze hooks and recomputes caption+image RoPE positional embeddings on each stage transition. > **Note:** Always specify `--height 1024 --width 1024` (or another resolution where H\_lat and W\_lat are both divisible by 2). Z-Image's default resolution (360×640) produces a 45×80 latent where H=45 is not divisible by the patch size. ### Usage ```bash theme={null} # Standard fullres — unchanged behavior sglang generate --model-path Tongyi-MAI/Z-Image \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 # Progressive dct_rewind L1 δ=0.10 → 2.33× denoising speedup sglang generate --model-path Tongyi-MAI/Z-Image \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 50 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.10 ``` ### Choosing delta | δ | Coarse steps (50 total) | Denoising speedup | | ------ | ----------------------- | ----------------- | | `0.01` | 26 @ 64² + 24 @ 128² | **1.53×** | | `0.05` | 35 @ 64² + 15 @ 128² | **2.03×** | | `0.10` | 42 @ 64² + 8 @ 128² | **2.33×** | Z-Image achieves higher progressive speedups than FLUX.1 at the same δ because it uses dual CFG (two forward passes per step), doubling the absolute attention savings at coarse resolution. `0.10` is the recommended tradeoff. ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="Tongyi-MAI/Z-Image", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 50, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.10, }) ``` *** ## Qwen-Image Qwen-Image uses the same 2×2 patchify convention as FLUX.1 (in\_channels=64, C=16), so the same progressive stage wires in with model-specific hooks for RoPE (`freqs_cis`) and spatial metadata (`img_shapes`). ```bash theme={null} # Standard fullres — unchanged behavior sglang generate --model-path Qwen/Qwen-Image \ --prompt "A serene mountain lake at golden hour" # Progressive dct_rewind L1 δ=0.20 → 1.69× denoising speedup sglang generate --model-path Qwen/Qwen-Image \ --prompt "A serene mountain lake at golden hour" \ --progressive-mode dct_rewind --progressive-levels 1 --progressive-delta 0.20 \ --num-inference-steps 30 --dit-cpu-offload false ``` Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only. | Config | Stage split | Denoise | Speedup | | --------------------- | ------------------ | ------- | --------- | | Fullres (baseline) | 30 @ 128² | 43.00 s | 1.00× | | dct\_rewind L1 δ=0.05 | 13\@64² + 17\@128² | 33.25 s | **1.29×** | | dct\_rewind L1 δ=0.10 | 16\@64² + 14\@128² | 33.86 s | **1.27×** | | dct\_rewind L1 δ=0.20 | 19\@64² + 11\@128² | 25.40 s | **1.69×** | ## Ideogram 4 Supports `ideogram-ai/ideogram-4`. Ideogram 4 uses a **dual-transformer architecture**: a conditional transformer (text + image tokens) and a separately-weighted unconditional transformer (image tokens only, zero LLM features). Both transformers shrink at coarse resolution, providing the same token-ratio benefit as single-transformer models. > **Note:** Ideogram 4's logit-normal noise schedule (`std=1.75`, `mu=0`) concentrates steps near the mid-sigma range. Fewer steps fall in the high-sigma coarse-eligible region compared to FLUX, which limits the achievable speedup at a given δ. ### Usage **20-step (V4\_DEFAULT\_20 preset)** ```bash theme={null} sglang generate \ --model-path ideogram-ai/ideogram-4 \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 20 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` **48-step (V4\_QUALITY\_48 preset)** ```bash theme={null} sglang generate \ --model-path ideogram-ai/ideogram-4 \ --prompt "A serene mountain lake at golden hour, photorealistic" \ --height 1024 --width 1024 \ --num-inference-steps 48 \ --dit-cpu-offload false \ --progressive-mode dct_rewind \ --progressive-levels 1 \ --progressive-delta 0.05 ``` ### Benchmark Hardware: RTX A6000 48 GB, `torch_sdpa`, `--dit-cpu-offload false`. Timing = denoising loop only. **20-step (V4\_DEFAULT\_20)** | Config | Stage split | Denoise | Speedup | | --------------------- | ------------------ | ------- | --------- | | Fullres (baseline) | 20 @ 64² | 53.99 s | 1.00× | | dct\_rewind L1 δ=0.01 | 6 @ 32² + 14 @ 64² | 43.47 s | **1.24×** | | dct\_rewind L1 δ=0.05 | 9 @ 32² + 11 @ 64² | 38.14 s | **1.42×** | | dct\_rewind L1 δ=0.10 | 11 @ 32² + 9 @ 64² | 34.60 s | **1.56×** | **48-step (V4\_QUALITY\_48)** | Config | Stage split | Denoise | Speedup | | --------------------- | ------------------- | -------- | --------- | | Fullres (baseline) | 48 @ 64² | 130.92 s | 1.00× | | dct\_rewind L1 δ=0.01 | 12 @ 32² + 36 @ 64² | 109.79 s | **1.19×** | | dct\_rewind L1 δ=0.05 | 21 @ 32² + 27 @ 64² | 93.83 s | **1.40×** | | dct\_rewind L1 δ=0.10 | 26 @ 32² + 22 @ 64² | 84.94 s | **1.54×** | ### Python API ```python theme={null} from sglang.multimodal_gen import DiffGenerator gen = DiffGenerator.from_pretrained( model_path="ideogram-ai/ideogram-4", dit_cpu_offload=False, ) result = gen.generate(sampling_params_kwargs={ "prompt": "A serene mountain lake at golden hour, photorealistic", "num_inference_steps": 48, "height": 1024, "width": 1024, "progressive_mode": "dct_rewind", "progressive_levels": 1, "progressive_delta": 0.05, }) ``` *** ## Limitations * **Sequence parallelism incompatible.** Cannot be combined with `--ulysses-degree` or `--ring-degree`. The stage raises a `RuntimeError` if SP is enabled. * **torch.compile incompatible.** Compiled kernels have a fixed sequence length; the resolution transition causes a recompile or error. Use progressive without `--enable-torch-compile`. * **Cache-DiT interaction is experimental.** The stage refreshes Cache-DiT context at resolution transitions, but quality and speedup should be benchmarked before relying on this combination. ## References * [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736) * [SGLang Diffusion Performance Optimization](./performance-optimization) # Quantization Source: https://docs.sglang.io/docs/sglang-diffusion/quantization SGLang-Diffusion treats component path selection and quantized checkpoint materialization as separate capabilities. Every loaded component can use an independent checkpoint path, but that checkpoint is quantized only when its selected loader supports the serialized format. Transformer, encoder, and VAE precision are resolved independently. ## Quick Reference Use these paths: * `--model-path`: the base or original model * `--component-paths.` / `---path`: replace a component from `model_index.json` or a native registered module with an independent repo/local directory; for transformer and native encoder loaders, an explicit weight filename retains the base component config * `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json` * `--transformer-weights-path`: replacement transformer weights in safetensors format (file, directory, or Hub repository/file) or a supported GGUF file (local or Hub) * `--quantization`: override the quantization method used by the transformer loader * `--quantization-ignored-layers`: transformer layer name patterns to keep unquantized during online quantization (e.g. `attention.to_`) * `--component-quantizations.`: explicitly apply supported online quantization to an unquantized component * `--component-quantization-ignored-layers.`: component-local layer patterns to keep unquantized during that online quantization * `--component-paths.text_encoder`: replace a native text encoder with a checkpoint whose `quantization_config` is auto-detected * `--text-encoder-path`: shorter alias for `--component-paths.text_encoder` * `--kv-cache-quant`: compress completed causal KV-cache chunks for supported realtime models Recommended example for pre-quantized checkpoints: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "a curious pikachu" ``` For quantized transformers-style transformer component folders: ```bash theme={null} sglang generate \ --model-path /path/to/base-model \ --transformer-path /path/to/quantized-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" ``` NOTE: Some model-specific integrations also accept a quantized repo or local directory directly as `--model-path`, but that is a compatibility path. If a repo contains multiple candidate checkpoints, pass `--transformer-weights-path` explicitly. MiniMax-H3 is a verified example for Comfy safetensors with per-layer metadata, including `pruned_fp8_scaled` and serialized ConvRot INT8. Other Comfy FP8 exports are also auto-detected: the presence of an input scale selects static activation scaling, while its absence selects dynamic scaling. Pass one selected FL2VA or Ref2VA DiT file by local path, `owner/repo/path/file.safetensors`, or direct Hugging Face file URL; do not combine it with `--quantization`. Its GGUF usage is documented in the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#pre-quantized-gguf-transformer). ## Quantized Component Repositories Every loaded component can point to an independent repository, but path routing does not imply that every loader can materialize every quantization format. SGLang resolves quantized component checkpoints through one of three explicit paths: * load through an SGLang quantization implementation; * delegate a standard component to Transformers or Diffusers; * fail closed in a native plain-state loader when the format cannot be restored. | Component path | Quantized checkpoint behavior | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. | | `text_encoder*`, `image_encoder*` | Passes detected metadata to the registered native encoder. Loading continues only when that implementation constructs compatible quantized layers; unknown or incompatible combinations fail closed. | | `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. | | Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. | | Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. | `--quantization` is the explicit override for the primary transformer loader; `--component-quantizations.` expresses the same intent for one supported component. Pair the latter with `--component-quantization-ignored-layers.` to keep matching layers unquantized. Pre-quantized component repositories instead select their format through their own metadata and the capability of the selected loader. A generic string override without a matching materialization backend would advertise support that the component does not have, while a quantized weight file without matching config metadata cannot be identified or restored generically. ## Quant Families Here, `quant_family` means a checkpoint and loading family with shared CLI usage and loader behavior. It is not just the numeric precision or a kernel backend.
quant\_family checkpoint form canonical CLI supported models extra dependency platform / notes
fp8 / mxfp4 (online quantization) Unquantized checkpoint (offline via AMD Quark coming soon) --quantization \{fp8,mxfp4} Z-Image-Turbo (validated), others likely work. More support coming soon. MXFP4: aiter on ROCm MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to fp8 / mxfp4 dynamically.
kitchen\_int8 (online quantization) Unquantized BF16/FP16 checkpoint --quantization kitchen\_int8 MiniMax-H3 (validated on 1× RTX 4090 24 GB) comfy-kitchen Data-free INT8 ConvRot at load time via comfy\_kitchen.int8\_linear. Layers whose input dim is not divisible by the group size stay in BF16. Requires Turing+ (SM75).
fp8 (offline quantization) Quantized transformer component folder, or safetensors with quantization\_config metadata --transformer-path or --transformer-weights-path Native DiTs whose linear layers support the selected FP8 method; validate quality per model None Component-folder and single-file flows are both supported
modelopt-fp8 Converted ModelOpt FP8 transformer directory or repo with config.json --transformer-path FLUX.1, FLUX.2, Wan2.2, HunyuanVideo, Qwen Image, Qwen Image Edit None Serialized config stays quant\_method=modelopt with quant\_algo=FP8; dit\_layerwise\_offload is supported and dit\_cpu\_offload stays disabled
auto-round W4A16 Transformer component repo with a self-describing quantization\_config and auto\_round:auto\_gptq packing --transformer-path Native dense DiTs with compatible component parameter mappings; MiniMax-H3 Diffusers components are supported None Auto-detected; reuses the SRT GPTQ/Marlin backend. No --quantization flag is needed; use TP/sequence parallelism rather than FSDP.
modelopt-nvfp4 Mixed transformer directory/repo with config.json, raw or Comfy layer-marked NVFP4 safetensors, or full ModelOpt Diffusers repo --transformer-path for mixed overrides; --transformer-weights-path for raw exports; --model-path for full repos FLUX.1, FLUX.2, Wan2.2, Qwen Image, Qwen Image 2512, Qwen Image Edit, Qwen Image Edit 2511, MiniMax-H3 None Mixed override repos keep the base model separate; full Qwen Image exports can be loaded directly as --model-path; raw exports such as black-forest-labs/FLUX.2-dev-NVFP4 use the weights-path flow. Comfy markers select NVFP4 plus INT8 or FP8 companion linears automatically; omit --quantization.
gguf One selected GGUF DiT or native encoder file --transformer-weights-path, or an explicit encoder file through --component-paths.\ MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs and its Qwen3-VL text encoder None CUDA only and auto-detected; supports standard and K-quant GGML types. TP requires aligned GGML blocks; component/layerwise offload is supported and FSDP is not.
comfy-fp8 One selected Comfy safetensors file with per-layer comfy\_quant metadata --transformer-weights-path MiniMax-H3 pruned FL2VA / Ref2VA DiTs None CUDA; auto-detected; TP, sequence parallelism, and component/layerwise offload are supported, while FSDP is not. Checkpoint-marked fc2 layers retain FP8 storage and use compute-dtype matmul.
comfy-int8-convrot One selected safetensors file with per-layer int8\_tensorwise and ConvRot metadata --transformer-weights-path, or an explicit weight file through --component-paths.\ Native DiTs and encoders whose parameter mappings preserve each marked linear; MiniMax-H3 DiT and Qwen3-VL encoder checkpoints have validated tensor contracts comfy-kitchen CUDA; auto-detected; uses the fused Kitchen INT8 kernel and validates weight/scale layout before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary. The H3 256-group DiT supports TP1/2/4, not TP8; its Qwen3-VL encoder keeps TP8 by replicating only incompatible row projections. Offload is supported; FSDP is not.
mxfp8 Self-describing serialized weights, or BF16/FP16 weights for online quantization --transformer-weights-path, or --quantization mxfp8 for online quantization Native diffusion linears; mixed MiniMax-H3 checkpoints are selected per layer SRT's platform MXFP8 backend Serialized metadata is auto-detected. NVIDIA and ROCm reuse SRT's dense MXFP8 kernels; Ascend keeps its native online path. Mixed per-layer checkpoints do not support FSDP.
comfy-w4a8-convrot Safetensors with serialized asym\_w4a8\_int8 layer metadata and packed weights --transformer-weights-path or --component-paths.text\_encoder MiniMax-H3 FL2VA / Ref2VA DiTs and its native Qwen3-VL encoder comfy-kitchen>=0.2.27 Auto-detected; omit --quantization. Requires SM80+ and validates packed weights, group/channel scales, and optional codebooks before model construction. Mixed encoder files may keep their embedding tensorwise INT8. TP must preserve ConvRot group boundaries; offload is supported and FSDP is not.
comfy-w4a4-convrot Safetensors with serialized convrot\_w4a4 metadata, optionally mixed with int8\_tensorwise layers --transformer-weights-path or --component-paths.text\_encoder Native DiTs and encoders with matching parameter mappings; MiniMax-H3 FL2VA / Ref2VA DiTs and Qwen3-VL encoder layouts are recognized comfy-kitchen Auto-detected; omit --quantization. Each layer dispatches to its serialized W4A4 or INT8 ConvRot kernel. CUDA requires SM75+; TP must preserve each format's quantization and ConvRot group boundaries. Offload is supported and FSDP is not.
quanto-int8 One native encoder safetensors file with an embedded Quanto quantization map An explicit weight file through --component-paths.\ Native encoders whose mapped linear layers consume every declared qint8 entry; MiniMax-H3's Qwen3-VL encoder is supported None Auto-detected weight-only qint8 storage. Each active matrix is dequantized to the compute dtype for the ordinary linear operation, so this reduces stored/resident weight memory rather than promising INT8 GEMM speed. TP and offload are supported; FSDP is not.
qvg-kv Unquantized model with runtime causal KV-cache compression --kv-cache-quant \{int4,int2} LingBot World realtime causal path quant-videogen CUDA only; compresses completed cache chunks rather than model weights; lossy and disabled by default
nunchaku-svdq Pre-quantized Nunchaku transformer weights, usually named svdq-\{int4|fp4}\_r\{rank}-... --transformer-weights-path Model-specific support such as Qwen-Image, FLUX, and Z-Image nunchaku SGLang can infer precision and rank from the filename and supports both int4 and nvfp4
msmodelslim Pre-quantized msmodelslim transformer weights --model-path Wan2.2 family None Currently only compatible with the Ascend NPU family and supports mxfp8, mxfp4, w8a8, and w4a4
gguf A single community .gguf holding a transformer or native encoder --transformer-weights-path or --component-paths.\ MiniMax-H3 fl2va DiT (original and pruned AdaLN curve) and Qwen3-VL text encoder None CUDA only, no FSDP. TP requires GGML-aligned shard boundaries. Packed weights reduce checkpoint and resident-weight memory; they are dequantized per use, so this is not a speed claim. See GGUF.
## Causal KV-Cache Quantization Quant-VideoGen KV-cache quantization targets long-running autoregressive video sessions, where the causal self-attention cache can become comparable to the model weights. It does not change or quantize the checkpoint weights. See [Realtime and Causal Video Models](./realtime_models) for the session lifecycle, supported pipelines, and the distinction between realtime and request-based causal generation. Install the optional dependency without allowing its stale Torch requirement to replace SGLang's pinned Torch version, then enable int4 compression when serving a supported LingBot World realtime pipeline: ```bash theme={null} pip install "sglang[diffusion,diffusion-qvg]" pip install --no-deps quant-videogen==0.1.0 sglang serve \ --model-path robbyant/lingbot-world-fast-diffusers \ --pipeline-class-name LingBotWorldCausalDMDPipeline \ --num-gpus 4 \ --ulysses-degree 4 \ --kv-cache-quant int4 \ --dit-cpu-offload false \ --text-encoder-cpu-offload false ``` ### Storage Policy The current chunk is rewritten at every denoising step, so it remains in BF16. The newest `--kv-cache-quant-keep-recent` completed chunks also remain in BF16. Older completed chunks are stable and are packed once with Progressive Residual Quantization (PRQ); their dense BF16 tensors are then released. When a transformer layer runs attention, its packed visible chunks are dequantized and concatenated with the recent BF16 chunks. This creates one layer's dense attention view at a time instead of keeping dense windows resident for every transformer layer. ### How PRQ Works For each K or V vector, PRQ uses k-means to select a centroid, then quantizes the remaining error: ```text theme={null} x = centroid_1 + residual_1 residual_1 = centroid_2 + residual_2 ... x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual) ``` Each additional stage applies another centroid lookup to the previous stage's residual. SGLang's default uses one stage, 128 centroids, and an int4 or int2 block-quantized residual. More stages or centroids can reduce reconstruction error but add codebook storage and packing work. PRQ is the compression algorithm; selecting older completed chunks is the runtime storage policy that makes it practical. Stable chunks are compressed once, while mutable and recent chunks avoid repeated packing and retain higher precision. ### Quality And Performance KV-cache quantization is lossy. Disabling it uses the original dense BF16 cache and is bit-exact with the unmodified path. Enabling int4 or int2 reconstructs an approximation of K and V, so fixed-seed generated frames are not expected to be pixel-identical to BF16. In the initial LingBot measurements, int4 used about 47% of the dense resident KV-cache memory for a 24-frame window and added about 18% per-chunk latency. Int2 used about 37% of the dense resident KV-cache memory but introduces more quantization error. These measurements are configuration-specific; benchmark memory, latency, temporal consistency, identity stability, and motion quality on the intended session length. Start with int4 unless capacity requires int2. The current implementation is limited to the LingBot realtime sliding-window-and-sink path, including Ulysses sequence sharding. It does not support LongLive2 pinned sinks, global sinks, or dynamically growing caches. ### Tuning Options | Option | Default | Effect | | ---------------------------------- | -------: | ---------------------------------------------------------------- | | `--kv-cache-quant {off,int4,int2}` | `off` | Enables QVG KV-cache compression and selects residual precision. | | `--kv-cache-quant-stages` | `1` | Number of progressive centroid-residual stages. | | `--kv-cache-quant-centroids` | `128` | Number of k-means centroids per stage. | | `--kv-cache-quant-block-size` | `64` | Block size used to quantize the final residual. | | `--kv-cache-quant-iters` | `2` | K-means iterations used while packing a chunk. | | `--kv-cache-quant-asymmetric` | disabled | Uses asymmetric residual quantization. | | `--kv-cache-quant-keep-recent` | `1` | Number of newest completed chunks retained in BF16. | | `--kv-cache-quant-sink {0,1}` | `1` | Whether to quantize completed sink chunks. | | `--kv-cache-quant-sink-keep` | `0` | Number of leading sink chunks retained in BF16. | ## Online Quantization This section describes the online methods currently implemented by the transformer loader. They are useful when a pre-quantized transformer checkpoint is not available. Encoders, VAEs, and auxiliary components are independent: their repositories may still carry serialized quantized weights, which are restored when the selected component loader supports that format. ### FP8 Online Quantization Apply FP8 quantization to a supported unquantized DiT checkpoint: ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization fp8 \ --prompt "a beautiful sunset" \ --save-output ``` MiniMax-H3 supports this path while preserving its required FP32 patch, timestep, and output projections. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) for its distributed serving recipe. ### MXFP4 Online Quantization MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.** ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization mxfp4 \ --prompt "a beautiful sunset" \ --save-output ``` **Note:** Requires `aiter` package with MXFP4 kernel support ### Kitchen INT8 Serialized Comfy ConvRot INT8 DiTs use `--transformer-weights-path`; compatible native encoders use an explicit file through `--component-paths.` or its component alias. Both are auto-detected from per-layer markers and load INT8 weights and row scales directly; omit `--quantization`. For a BF16 checkpoint, `--quantization kitchen_int8` instead performs online quantization after loading: `kitchen_int8` quantizes DiT linear weights online from the stock BF16 checkpoint. Forward uses the fused `comfy_kitchen.int8_linear` op (rotation, dynamic per-row activation quantization, INT8 GEMM, dequant, and bias). Install the optional dependency first: ```bash theme={null} pip install comfy-kitchen ``` ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --quantization kitchen_int8 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Quantization runs after the model's weight loaders, so MiniMax-H3's grouped `qkv` reorder is already applied. Layers whose input dim is not divisible by the group size (256) stay in BF16 instead of failing the load; H3's AdaLN projections take that path. `kitchen_int8` is approximate and is not a consistency ground-truth mode. The BF16 path is unchanged when `comfy-kitchen` is not installed. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) for the 24 GB offload recipe, including why `vae` must stay out of `--layerwise-offload-components`. Large-M GEMMs (`rows > 8192` and `out_features >= 8192`) are split by rows so the fused kernel stays on the data-parallel CUTLASS config. Override the thresholds with `SGLANG_KITCHEN_INT8_MAX_ROWS` and `SGLANG_KITCHEN_INT8_MIN_SPLIT_N`. ### Skipping Layers By default, transformer online quantization quantizes every supported linear layer in that component. However, `--quantization-ignored-layers` can keep specific transformer layers in their original precision: ```bash theme={null} sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization fp8 \ --quantization-ignored-layers attention.to_ \ --prompt "a beautiful sunset" \ --save-output sglang generate \ --model-path Tongyi-MAI/Z-Image-Turbo \ --quantization mxfp4 \ --quantization-ignored-layers attention.to_ \ --prompt "a beautiful sunset" \ --save-output ``` Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns. ## Transformers Component BnB4 Model components that already have a native Transformers loading path can load serialized BitsAndBytes 4-bit checkpoints with a standard top-level `quantization_config`. Plain checkpoints keep using an available native SGLang implementation. For example, replace FLUX's T5 component with the official Diffusers checkpoint: ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.1-dev \ --component-paths.text_encoder_2 \ diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2 ``` The quantized component must stay resident. SGLang rejects component or layerwise offload, nonstandard metadata locations, and native-only component fallbacks for this path instead of silently changing the checkpoint contract. Diffusion DiT components declared under the Diffusers library use the separate quantization backends documented above. ## Validated ModelOpt Checkpoints This section is the canonical support matrix for the thirteen published diffusion ModelOpt checkpoints currently wired up in SGLang docs and validation coverage. Published checkpoints keep the serialized quantization config as `quant_method=modelopt`; the FP8 vs NVFP4 split below is a documentation label derived from `quant_algo`. Twelve of the thirteen repos live under `lmsys/*`. The FLUX.2 NVFP4 entry keeps the official `black-forest-labs/FLUX.2-dev-NVFP4` repo.
Quant Algo Base Model Preferred CLI HF Repo Current Scope Notes
FP8 black-forest-labs/FLUX.1-dev --transformer-path lmsys/flux1-dev-modelopt-fp8-sglang-transformer single-transformer override, deterministic latent/image comparison, H100 benchmark, torch-profiler trace SGLang converter keeps a validated BF16 fallback set for modulation and FF projection layers; use --model-id FLUX.1-dev for local mirrors
FP8 black-forest-labs/FLUX.2-dev --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer single-transformer override load and generation path published SGLang-ready transformer override
FP8 Wan-AI/Wan2.2-T2V-A14B-Diffusers --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer primary transformer quantized, transformer\_2 kept BF16 primary-transformer-only path; keep transformer\_2 on the base checkpoint, and do not describe this as dual-transformer full-model FP8 unless that path is validated separately
FP8 hunyuanvideo-community/HunyuanVideo --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer single-transformer override, BF16-vs-FP8 video comparison, H100 benchmark, torch-profiler trace HunyuanVideo uses different ModelOpt/diffusers and SGLang runtime module names; the converter maps those names before writing FP8 scale tensors and BF16 fallback ignores
FP8 Qwen/Qwen-Image --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer single-transformer override, BF16-vs-FP8 image comparison, H100 benchmark, torch-profiler trace shares the Qwen Image FP8 fallback preset; keep img\_in, txt\_in, timestep embedder, norm\_out.linear, proj\_out, img\_mod/txt\_mod, and img\_mlp.net.2 in BF16
FP8 Qwen/Qwen-Image-Edit-2511 --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer TI2I edit path, BF16-vs-FP8 image comparison, H100 benchmark shares QwenImageTransformer2DModel with Qwen Image and uses the same Qwen Image FP8 fallback preset
NVFP4 black-forest-labs/FLUX.1-dev --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer mixed BF16+NVFP4 transformer override, correctness validation, 4x RTX 5090 benchmark, torch-profiler trace use build\_modelopt\_nvfp4\_transformer.py; validated builder keeps selected FLUX.1 modules in BF16 and sets swap\_weight\_nibbles=false
NVFP4 black-forest-labs/FLUX.2-dev --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 packed-QKV load path official raw export repo; validated packed export detection and runtime layout handling
NVFP4 Wan-AI/Wan2.2-T2V-A14B-Diffusers --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer primary transformer quantized with ModelOpt NVFP4, transformer\_2 kept BF16 primary-transformer-only path; keep transformer\_2 on the base checkpoint; the default FP4 GEMM backend is flashinfer\_trtllm
NVFP4 Qwen/Qwen-Image --model-path lmsys/qwen-image-modelopt-nvfp4-sglang full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly; exported with ModelOpt PR #1706 SVDQuant NVFP4 (--format fp4, max calibration, block size 16) and BF16 fallbacks for attention-sensitive modules plus first/last transformer blocks
NVFP4 Qwen/Qwen-Image-2512 --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison, B200 CI case same full-repo loader path as Qwen Image; this is the Qwen Image NVFP4 representative in multimodal-gen-test-1-b200
NVFP4 Qwen/Qwen-Image-Edit --model-path lmsys/qwen-image-edit-modelopt-nvfp4-sglang TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe
NVFP4 Qwen/Qwen-Image-Edit-2511 --model-path lmsys/qwen-image-edit-2511-modelopt-nvfp4-sglang TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe
These thirteen checkpoints are the intended ModelOpt documentation support set. The B200 diffusion CI job (`multimodal-gen-test-1-b200`) uses a representative NVFP4 subset and includes `lmsys/qwen-image-2512-modelopt-nvfp4-sglang` for Qwen Image coverage. ## ModelOpt FP8 ### Usage Examples Converted ModelOpt FP8 transformer repos should be loaded as transformer component overrides. If the repo or local directory already contains `config.json`, use `--transformer-path`. Full Diffusers repos such as the NVIDIA Wan2.2 FP8 checkpoint can be passed directly with `--model-path`. ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \ --prompt "a fox walking through neon rain" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path hunyuanvideo-community/HunyuanVideo \ --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer \ --height 544 --width 960 --num-frames 17 \ --prompt "A cinematic shot of a red sports car driving through rain at night" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image-Edit-2511 \ --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer \ --image-path /path/to/input.png \ --prompt "Turn the scene into a warm watercolor illustration" \ --save-output ``` ### Notes * `--transformer-path` is the canonical flag for converted ModelOpt FP8 transformer component repos or directories that already carry `config.json`. * If the override repo or local directory contains its own `config.json`, SGLang reads the quantization config from that override instead of relying on the base model config. * `--transformer-weights-path` still works when you intentionally point at raw weight files or a directory that should be metadata-probed as weights first. * `dit_layerwise_offload` is supported for ModelOpt FP8 checkpoints. * `dit_cpu_offload` still stays disabled for ModelOpt FP8 checkpoints. * The layerwise offload path now preserves the non-contiguous FP8 weight stride expected by the runtime FP8 GEMM path. * On disk, the quantization config stays `quant_method=modelopt` with `quant_algo=FP8`; the `modelopt-fp8` label in this document is a support family name, not a serialized config key. * To build the converted checkpoint yourself from a ModelOpt diffusers export, use `python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer`. ## ModelOpt NVFP4 ### Usage Examples For mixed ModelOpt NVFP4 transformer overrides that already contain `config.json`, keep the base model and quantized transformer separate and use `--transformer-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.1-dev \ --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` For raw NVFP4 exports such as the official FLUX.2 release, use `--transformer-weights-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev \ --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` SGLang also supports passing the NVFP4 repo or local directory directly as `--model-path`: ```bash theme={null} sglang generate \ --model-path black-forest-labs/FLUX.2-dev-NVFP4 \ --prompt "A Logo With Bold Large Text: SGL Diffusion" \ --save-output ``` For a dual-transformer Wan2.2 export where only the primary `transformer` was quantized: ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \ --prompt "a fox walking through neon rain" \ --save-output ``` For full Qwen Image NVFP4 exports, load the published repo directly: ```bash theme={null} sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` For high-resolution Qwen-Image-family generations on B200, the FlashInfer CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM backend: ```bash theme={null} SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \ sglang generate \ --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \ --width 2048 --height 2048 \ --prompt "A tiny astronaut reading a book under a glass greenhouse" \ --save-output ``` ### Notes * Use `--transformer-path` for mixed ModelOpt NVFP4 transformer repos or local directories that already include `config.json`. * Use `--transformer-weights-path` for raw NVFP4 exports, individual safetensors files, or repo layouts that should be treated as weights first. * For dual-transformer pipelines such as `Wan2.2-T2V-A14B-Diffusers`, the primary `--transformer-path` override targets only `transformer`. Use a per-component override such as `--transformer-2-path` only when you intentionally want a non-default `transformer_2`. * On Blackwell, the diffusion ModelOpt NVFP4 path defaults to FlashInfer TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`). * The published Qwen Image NVFP4 exports keep the `img_mod`/`txt_mod` modulation projections and first/last transformer blocks in BF16. * Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the validated ModelOpt exports were faster than BF16 at 2048x2048 with `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass`, while 1024x1024 remained BF16-faster. * Direct `--model-path` loading is the canonical path for full Qwen Image ModelOpt NVFP4 repos and a compatibility path for FLUX.2 NVFP4-style repos or local directories. * If `--transformer-weights-path` is provided explicitly, it takes precedence over the compatibility `--model-path` flow. * For local directories, SGLang first looks for `*-mixed.safetensors`, then falls back to loading from the directory. * To force the diffusion ModelOpt FP4 path onto a different FlashInfer backend, set `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`. Supported values include `flashinfer_cudnn`, `flashinfer_cutlass`, and `flashinfer_trtllm`. * On disk, the quantization config stays `quant_method=modelopt` with `quant_algo=NVFP4`; the `modelopt-nvfp4` label here is again a documentation family name rather than a serialized config key. ## GGUF GGUF loads a community-quantized transformer from a single `.gguf` file while the rest of the pipeline — VAE, text encoder, scheduler, tokenizer — keeps loading from the base model. GGUF primarily reduces checkpoint, host-memory, and resident-weight size. For example, MiniMax-H3's transformer is 17.5 GiB as Q4\_K\_M versus 61.7 GiB as BF16. With full layerwise offload, VAE decode and offload buffers can still dominate peak GPU memory, but each streamed DiT layer also transfers fewer bytes. Packed linears reuse SRT's GGUF type definitions and CUDA dequantization, then run the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM regime and are slower at diffusion sequence lengths. GGUF remains a capacity-oriented option; latency depends on the quantization type, activation shape, and placement policy. Layers the checkpoint stores unquantized (F32/F16/BF16) take the ordinary linear path rather than the packed one. Their precision is then whatever the model declares for that layer, exactly as on the safetensors path — a checkpoint cannot raise a layer above the model's own dtype by storing it wider. For MiniMax-H3 the two agree: the layers it pins to FP32 are the ones the validated checkpoint leaves unquantized, and `post_load_weights` fails the load if any of them ends up narrower. ### Usage No extra install: `gguf` is already a core SGLang dependency. `--model-path` stays the base model; `--transformer-weights-path` takes the GGUF. A local path, `owner/repo/file.gguf`, or `owner/repo:QUANT_TYPE` all work. The quant-type shorthand is accepted only when exactly one repository file matches it; otherwise SGLang lists the candidates and asks for a full path. ```bash theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \ --num-gpus 1 \ --attention-backend fa \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --dit-offload-prefetch-size 1 \ --dit-layerwise-resident-layers 0 \ --enable-torch-compile false \ --port 30010 ``` Note that `--quantization gguf` is not the selector — the quantization is read from the file itself, so passing the file is what enables the path. MiniMax-H3 GGUF checkpoints may use either the original timestep MLP or the pruned AdaLN curve architecture. Repositories often contain both FL2VA and Ref2VA files, so use the full Hub file reference instead of an ambiguous `owner/repo:QUANT_TYPE` selector: ```bash theme={null} sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --transformer-weights-path \ unsloth/MiniMax-H3-GGUF/minimax_h3_fl2va_pruned-Q4_K.gguf \ --performance-mode memory \ --layerwise-offload-components dit,text_encoder \ --enable-torch-compile false \ --port 30010 ``` The pruned architecture keeps its sampled curve and reduced AdaLN projections in FP32, matching the published checkpoint implementation. This precision island deliberately bypasses the BF16-only fused modulation kernels. #### Whether to offload the VAE Adding `vae` to `--layerwise-offload-components` trades a lot of latency for some peak VRAM, because the video VAE decoder is re-streamed per decode tile. Measured on 1× RTX 5090 with this checkpoint, 1344×768 × 107 frames: | `--layerwise-offload-components` | Peak VRAM | Denoise | VAE decode | | -------------------------------- | -----------: | ------: | ---------: | | `dit,text_encoder` | 26.3 GiB | 39.0 s | **9.5 s** | | `dit,text_encoder,vae` | **19.6 GiB** | 39.0 s | 57.2 s | Denoise is unaffected, and the output is bit-identical either way. Leave the VAE resident unless the 6.7 GiB matters — on a 24 GB card by this measurement it does, and the 6× slower decode is the price of fitting. ### Constraints | Constraint | Reason | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | TP shard boundaries must align to GGML blocks | Column-parallel rows shard directly; row-parallel packed columns require each local input partition to contain whole quantization blocks | | No `--use-fsdp-inference` | FSDP does not preserve the GGUF packed-block layout | | CUDA only | The reused SRT GGML dequantization kernel currently ships for CUDA | | Native byte order only | A quantized block embeds its scales, so a non-native file cannot be byte-swapped as a whole | | No LoRA, and none of the H3 AdaLN cache flags | An adapter cannot be merged into packed blocks, and the AdaLN paths read the transformer's safetensors | | No `--quantization` | The checkpoint fixes the quantization; the flag would be a second, conflicting selector | Every constraint above fails at startup with an explanatory error rather than silently producing wrong output. Sequence parallelism (`--ulysses-degree` / `--ring-degree`) remains available because it shards activations rather than packed weights. TP is also available; startup rejects a degree that cuts a row-parallel matrix inside a GGML block. ### Validated scope | Model | Checkpoint | Hardware | Result | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------- | | MiniMax-H3 `fl2va` | [`leejet/MiniMax-H3-GGUF`](https://huggingface.co/leejet/MiniMax-H3-GGUF) `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x RTX 5090 (32 GiB) | t2va 1344x768, 107 frames, video + audio; 19.6-26.3 GiB peak depending on VAE offload | | MiniMax-H3 `fl2va` | `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 10.13 s and 17.17 GiB peak | | MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank | The DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight fidelity was checked tensor-by-tensor against the BF16 reference: cosine 1.00000 for the F32/BF16 tensors and 0.9973 for Q4\_K/Q4\_0. Not validated in the measurements above: any other quantization type, the `ref2va` partition, and a BF16-vs-GGUF output comparison. ## Nunchaku (SVDQuant) ### Install Install the runtime dependency first: ```bash theme={null} pip install nunchaku ``` For platform-specific installation methods and troubleshooting, see the [Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html). ### File Naming and Auto-Detection For Nunchaku checkpoints, `--model-path` should still point to the original base model, while `--transformer-weights-path` points to the quantized transformer weights. If the basename of `--transformer-weights-path` contains the pattern `svdq-(int4|fp4)_r{rank}`, SGLang will automatically: * enable SVDQuant * infer `--quantization-precision` * infer `--quantization-rank` Examples:
checkpoint name fragment inferred precision inferred rank notes
svdq-int4\_r32 int4 32 Standard INT4 checkpoint
svdq-int4\_r128 int4 128 Higher-quality INT4 checkpoint
svdq-fp4\_r32 nvfp4 32 fp4 in the filename maps to CLI value nvfp4
svdq-fp4\_r128 nvfp4 128 Higher-quality NVFP4 checkpoint
Common filenames:
filename precision rank typical use
svdq-int4\_r32-qwen-image.safetensors int4 32 Balanced default
svdq-int4\_r128-qwen-image.safetensors int4 128 Quality-focused
svdq-fp4\_r32-qwen-image.safetensors nvfp4 32 RTX 50-series / NVFP4 path
svdq-fp4\_r128-qwen-image.safetensors nvfp4 128 Quality-focused NVFP4
svdq-int4\_r32-qwen-image-lightningv1.0-4steps.safetensors int4 32 Lightning 4-step
svdq-int4\_r128-qwen-image-lightningv1.1-8steps.safetensors int4 128 Lightning 8-step
If your checkpoint name does not follow this convention, pass `--enable-svdquant`, `--quantization-precision`, and `--quantization-rank` explicitly. ### Usage Examples Recommended auto-detected flow: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \ --prompt "a beautiful sunset" \ --save-output ``` Manual override when the filename does not encode the quant settings: ```bash theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \ --enable-svdquant \ --quantization-precision int4 \ --quantization-rank 128 \ --prompt "a beautiful sunset" \ --save-output ``` ### Notes * `--transformer-weights-path` is the canonical flag for Nunchaku checkpoints. Older config names such as `quantized_model_path` are treated as compatibility aliases. * Auto-detection only happens when the checkpoint basename matches `svdq-(int4|fp4)_r{rank}`. * The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is written as `fp4`. * Lightning checkpoints usually expect matching `--num-inference-steps`, such as `4` or `8`. * Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) or SM12x GPUs. Hopper (SM90) is currently rejected. ## [ModelSlim](https://gitcode.com/Ascend/msmodelslim) MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. * **Installation** ```bash theme={null} # Clone repo and install msmodelslim: git clone https://gitcode.com/Ascend/msmodelslim.git cd msmodelslim bash install.sh ``` * **Multimodal\_sd quantization** Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card). > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). Run quantization using one-click quantization (recommended): ```bash theme={null} msmodelslim quant \ --model_path /path/to/wan2_2_float_weights \ --save_path /path/to/wan2_2_quantized_weights \ --device npu \ --model_type Wan2_2 \ --quant_type w8a8 \ --trust_remote_code True ``` For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo. > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim. * **Auto-Detection and different formats** For msmodelslim checkpoints, it's enough to specify only `--model-path`, the detection of quantization occurs automatically for each layer using parsing of `quant_model_description.json` config. In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format. For conversion, use the one-step `wan_repack.py` script: ```bash theme={null} python wan_repack.py \ --model-type Wan2.2-TI2V-5B \ --original-model-path {path_to_original_diffusers_model} \ --quant-path {path_to_quantized_model} \ --output-path {path_to_converted_model} ``` Supported `--model-type` values: `Wan2.2-TI2V-5B` (single-transformer), `Wan2.2-T2V-A14B` and `Wan2.2-I2V-A14B` (Cascade dual-transformer). The script automatically handles: copying the base model, converting quantized weights to Diffusers format, and restoring `config.json`. * **Usage Example** With auto-detected flow: ```bash theme={null} sglang generate \ --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \ --prompt "a beautiful sunset" \ --save-output ``` * **Available Quantization Methods**: * [x] `W4A4_DYNAMIC` linear with online quantization of activations * [x] `W8A8` linear with offline quantization of activations * [x] `W8A8_DYNAMIC` linear with online quantization of activations * [x] `W8A8_MXFP8` linear with offline quantization (msmodelslim pre-quantized weights) * [x] `mxfp8` linear with online quantization (`--quantization mxfp8`) * [x] `W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` linear with offline quantization (msmodelslim pre-quantized weights) * [x] `mxfp4_npu` linear with online quantization (`--quantization mxfp4_npu`) ## MXFP8 Quantization Self-describing serialized MXFP8 transformers are inferred from their checkpoint metadata. For online MXFP8 quantization, load the original FP16/BF16 model and add `--quantization mxfp8`. NVIDIA and ROCm reuse SRT's dense MXFP8 implementation; Ascend uses `npu_dynamic_mx_quant` and `npu_quant_matmul` with a block size of 32. ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp8 \ --prompt "a fox walking through neon rain" \ --save-output ``` The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On Ascend, the hardware requirement remains A5 series or newer; `npu_dynamic_mx_quant` is not available on A2/A3. ## MXFP8 Offline Quantization (msmodelslim) Pre-quantized MXFP8 weights exported by msmodelslim are auto-detected via `quant_model_description.json` (`W8A8_MXFP8` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load the converted model with `--model-path`: ```bash theme={null} sglang generate \ --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-mxfp8 \ --prompt "a beautiful sunset" \ --save-output ``` ## MXFP4 Online Quantization For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add `--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4` is reserved for the ROCm/aiter backend. Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses dual-level block scales with an L1 block size of 32 and an L0 block size of 512. ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --quantization mxfp4_npu \ --prompt "a fox walking through neon rain" \ --save-output ``` > **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_dual_level_mx_quant` > and `npu_dual_level_quant_matmul` are not available on A2/A3. > > **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim > flow uses pre-quantized weights and may produce different numerical results. ## MXFP4 Offline Quantization (msmodelslim) Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via `quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load the converted model with `--model-path`: ```bash theme={null} sglang generate \ --model-path {path_to_converted_mxfp4_model} \ --prompt "a beautiful sunset" \ --save-output ``` The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization, `mul_scale` is loaded and applied before activation quantization to keep activations aligned with the calibrated weights. # Realtime and Causal Video Models Source: https://docs.sglang.io/docs/sglang-diffusion/realtime_models Realtime and causal video pipelines generate video incrementally and reuse state across chunks. This differs from offline diffusion pipelines, which denoise one bounded latent sequence and release all request state when generation finishes. ## Execution Modes SGLang Diffusion exposes two related but distinct modes: | Mode | Lifetime | Interface | Examples | | ------------------------------- | --------------------------------------------------------------- | --------------------------------------- | ------------------------------------- | | Realtime session | State persists until the client disconnects or the session ends | `/v1/realtime_video/generate` WebSocket | LingBot World, SANA-WM realtime | | Request-based causal generation | State is reused across chunks within one request, then released | Standard video generation API | LongLive 2.0, batch-streaming SANA-WM | The realtime server retains model-specific state such as the causal self-attention KV cache, cross-attention cache, decoder history, and pending control events. State is isolated per session and is not reused by unrelated requests. A causal DiT is not automatically a realtime session model. The pipeline must also register a realtime adapter and implement the WebSocket session lifecycle. ## Supported Realtime Pipelines | Model family | Pipeline | Live controls | QVG KV-cache quantization | | ------------- | ------------------------------- | --------------------------------- | ------------------------- | | LingBot World | `LingBotWorldCausalDMDPipeline` | Camera actions and prompt updates | Supported | | SANA-WM | `SanaWMRealtimePipeline` | Camera actions | Not supported | Use the model cookbooks for launch commands, request schemas, and control-token details: * [LingBot World](/cookbook/diffusion/LingBot-World/LingBot-World) * [LingBot World 2.0](/cookbook/diffusion/LingBot-World/LingBot-World-2.0) * [SANA-WM](/cookbook/diffusion/SANA-WM/SANA-WM) For the complete model list, see [Supported Models and Optimization Compatibility](./compatibility_matrix). ## Causal Cache Controls Realtime requests can override two model defaults: * `realtime_causal_sink_size`: amount of stable prefix history retained as an attention sink * `realtime_causal_kv_cache_num_frames`: recent causal history retained in the rolling KV-cache window Larger windows preserve more history but increase resident memory and attention work. These fields are request/session controls; supported ranges and defaults remain model-specific. For supported LingBot World deployments, the server-level `--kv-cache-quant {off,int4,int2}` option compresses completed cache chunks. It is disabled by default and is lossy when enabled. Start with `int4`; use `int2` only when the additional memory reduction is worth the larger quality risk. See [Causal KV-Cache Quantization](./quantization#causal-kv-cache-quantization) for installation, storage policy, tuning options, memory/latency tradeoffs, and current limitations. QVG KV-cache quantization currently supports only the LingBot realtime sliding-window-and-sink path. It does not apply to SANA-WM realtime, LongLive 2.0 pinned sinks, global sinks, or dynamically growing caches. ## Deployment Considerations * Keep `--kv-cache-quant off` when bit-exact BF16 cache behavior is required. * Benchmark a representative session length. Short clips may not exercise cold-cache packing and can hide both its memory benefit and packing overhead. * Treat sequence parallelism as model-specific. Follow the model cookbook and the [Sequence Parallelism](./ring_sp_performance) guide instead of assuming one mesh is best for every realtime pipeline. * Realtime WebSocket clients must send an initialization message before control events. The exact MessagePack schema and output encoding are documented in each model cookbook. # Sequence Parallelism Source: https://docs.sglang.io/docs/sglang-diffusion/ring_sp_performance Sequence parallelism splits long image or video latent sequences across GPUs. In SGLang Diffusion, the public controls are: * `--sp-degree`: total sequence parallel degree * `--ulysses-degree`: Ulysses parallel degree * `--ring-degree`: ring parallel degree * `--sp-attention-mode`: attention exchange used inside each SP group The degrees must satisfy: ```text theme={null} sp_degree = ulysses_degree * ring_degree ``` The default `--sp-attention-mode ulysses` uses all-to-all to redistribute sequence shards over attention heads. `--sp-attention-mode kv_gather` keeps queries sequence-sharded and all-gathers keys and values, then computes each rank's local output directly. The K/V-gather mode currently supports non-causal attention with `--ring-degree 1`. Varlen calls through the legacy `UlyssesAttention` adapter and video sparse attention are not supported. Use SP when sequence length or video shape makes the DiT forward pass the bottleneck and the model supports sequence sharding. For latency-oriented multi-GPU Qwen/Wan deployments, also compare against CFG parallelism and FSDP; SP is not automatically the best multi-GPU setting for every model. ## Choosing The Attention Exchange
Mode Communication Memory Constraints
ulysses All-to-all before and after attention Full sequence with a shard of the attention heads during attention Attention head divisibility must match the Ulysses degree
kv\_gather All-gather K and V; Q and output remain sequence-sharded Replicates full K and V within the SP group Non-causal attention and ring\_degree=1; no legacy varlen or video sparse attention
Neither exchange is universally faster. K/V gather avoids the reverse all-to-all and can help when its local attention shape or collective is more efficient, while Ulysses can use less attention activation memory. Benchmark both on the target model, resolution, accelerator, and interconnect. For SP degree `P`, the approximate per-rank network payload of K/V gather relative to Ulysses is `P / 2`, excluding each rank's local shard. The payloads are therefore similar at SP2, while K/V gather moves about 2x as much data at SP4 and 4x at SP8. K/V gather may still be faster when all-gather and its local attention layout are more efficient, especially at low SP degrees, but this scaling makes the interconnect and input shape part of the selection policy. ## Recommended Commands ### Ulysses Sequence Parallelism The default mode needs only the total SP degree when ring parallelism is not used: ```bash theme={null} sglang serve \ --model-path Qwen/Qwen-Image \ --num-gpus 4 \ --sp-degree 4 \ --port 8898 ``` ### K/V-Gather Sequence Parallelism Use the same SP process-group layout and select the alternative attention exchange explicitly: ```bash theme={null} sglang serve \ --model-path Qwen/Qwen-Image \ --num-gpus 4 \ --sp-degree 4 \ --sp-attention-mode kv_gather \ --port 8898 ``` ### Tensor Plus Sequence Parallelism TP and SP use independent dimensions. With DP and CFG parallelism disabled, the required GPU count is `tp_size * sp_degree`. This example creates two TP groups across a two-rank SP dimension: ```bash theme={null} sglang serve \ --model-path Qwen/Qwen-Image \ --num-gpus 4 \ --tp-size 2 \ --sp-degree 2 \ --sp-attention-mode kv_gather \ --port 8898 ``` Omit `--sp-attention-mode kv_gather` to use TP plus Ulysses with the same `tp=2, sp=2` topology. #### How TP Plus SP Works TP and SP form orthogonal dimensions of the DiT process mesh. For `tp=2, sp=2`, ranks `[0, 1]` and `[2, 3]` are TP groups, while ranks `[0, 2]` and `[1, 3]` are SP groups. Each rank therefore belongs to one group of each type: * TP shards supported attention and MLP projection weights and computation, then communicates partial projection results inside the TP group. * SP shards the latent sequence and attention activations, then uses Ulysses or K/V gather inside the SP group. Pure SP replicates the DiT weights on every SP rank. Adding TP reduces the per-rank memory used by TP-sharded weights and keeps the sequence activation sharding from SP, at the cost of adding TP communication to every applicable DiT block. The exact memory reduction is model-dependent because not every parameter or runtime buffer is TP-sharded. TP plus SP should therefore be treated as a capacity and memory-latency Pareto option, not as the default latency winner. On a single NVSwitch node, pure SP often wins when the complete DiT weights fit on every GPU because it avoids the repeated TP collectives. Try TP plus SP when pure SP does not fit, when more memory headroom is required, or when its measured memory reduction is worth a small latency increase. The following representative eager results used eight H200 GPUs in one NVSwitch node. Times are median scheduler-side end-to-end latency. They illustrate the tradeoff rather than define a universal policy: | Model and workload | Fastest tested topology | TP plus SP Pareto point | Tradeoff | | ----------------------- | ---------------------------------------- | ----------------------------------------- | ------------------------------------ | | Qwen-Image, 1536x1536 | CFG2xSP4 Ulysses: 972.6 ms, 62.8 GiB/GPU | CFG2xTP2xSP2 K/V: 1017.4 ms, 48.4 GiB/GPU | 4.6% slower, 22.9% less peak memory | | Wan2.2-A14B, 832x480x81 | CFG2xSP4 K/V: 6573.5 ms, 61.9 GiB/GPU | CFG2xTP2xSP2 K/V: 7243.1 ms, 34.2 GiB/GPU | 10.2% slower, 44.7% less peak memory | | LTX2.3, 768x512x241 | SP8 K/V: 7258.2 ms, 55.4 GiB/GPU | TP2xSP4 K/V: 10372.3 ms, 37.8 GiB/GPU | 42.9% slower, 31.8% less peak memory | K/V gather can still improve TP plus SP at the same topology even when that topology is not the global latency winner. In the same experiment it improved TP2xSP4 by 4.2% for FLUX and 8.4% for LTX2.3, and improved CFG2xTP2xSP2 by 6.0% for Qwen-Image and 2.7% for Wan2.2-A14B, relative to Ulysses. Always compare the full candidate set, including pure SP, TP, CFG, and their feasible combinations, rather than selecting the SP attention backend first. ### FSDP Plus Sequence Parallelism FSDP can shard DiT weights across the same workers that participate in SP. Unlike TP times SP, the FSDP and SP degrees do not multiply the required GPU count. This is useful when pure SP is fast enough but replicated DiT weights or long-sequence activations leave too little memory headroom: ```bash theme={null} sglang serve \ --model-path Lightricks/LTX-2.3 \ --num-gpus 2 \ --use-fsdp-inference true \ --sp-degree 2 \ --sp-attention-mode kv_gather \ --port 8898 ``` FSDP adds weight all-gather communication, so compare it with pure SP when both fit. K/V gather has the same non-causal and `ring_degree=1` constraints under FSDP. ### Ring Sequence Parallelism This example uses two GPUs with `sp=2`, `ulysses=1`, and `ring=2`. ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --num-gpus 2 \ --sp-degree 2 \ --ulysses-degree 1 \ --ring-degree 2 \ --port 8898 ``` ### Single-GPU Baseline Use an explicit single-GPU baseline before attributing a gain to sequence parallelism. ```bash theme={null} sglang serve \ --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \ --num-gpus 1 \ --sp-degree 1 \ --ulysses-degree 1 \ --ring-degree 1 \ --port 8898 ``` ## Choosing The Degrees
Setting Typical use Notes
--sp-degree 1 Single-GPU or no sequence splitting Use this as the baseline.
--ulysses-degree N Ulysses-only sequence parallelism When ring parallelism is not needed, keep --ring-degree 1.
--ring-degree N Ring-based sequence splitting over long sequences Keep --sp-degree equal to ulysses\_degree \* ring\_degree.
## Cross-Node Sequence Parallelism Ulysses alone cannot scale sequence parallelism past the GPU count of one node: going wider either violates head-count divisibility or exposes an all-to-all across the slower inter-node link. Ring's point-to-point KV rotation is designed to overlap with attention compute, which tolerates a slower cross-node link far better than an all-to-all does — so the pattern for scaling SP across nodes is **node-local Ulysses × cross-node Ring**, not Ulysses alone. Cross-node launches add three flags on top of the usual SP degrees: * `--nnodes`: number of nodes. `--num-gpus` stays the *total* GPU count across every node; each node runs `num_gpus // nnodes` local workers. * `--node-rank`: this node's rank, `0` on the head node (which keeps the HTTP/TokenizerManager surface) and `1..nnodes-1` on the others (worker-only). * `--dist-init-addr`: a `host:port` rendezvous address reachable from every node — typically the head node's address. Run the same command on every node, changing only `--node-rank`: ```bash theme={null} # node 0 (head) sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 16 \ --nnodes 2 \ --node-rank 0 \ --dist-init-addr :23456 \ --sp-degree 16 \ --ulysses-degree 8 \ --ring-degree 2 \ --encoder-parallel replicate \ --port 30010 # node 1 (worker) sglang serve \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant ref2va \ --num-gpus 16 \ --nnodes 2 \ --node-rank 1 \ --dist-init-addr :23456 \ --sp-degree 16 \ --ulysses-degree 8 \ --ring-degree 2 \ --encoder-parallel replicate \ --port 30010 ``` `--encoder-parallel replicate` is required for cross-node deployments today: the `auto` fold decision is not yet node-boundary aware and will try to fold the text encoder across nodes, which crashes reference-conditioned encoders. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). Cross-node ring support is model-specific, not a property of the launch flags alone — see [Supported Models and Optimization Compatibility](/docs/sglang-diffusion/compatibility_matrix) for which models have it. Passing `--ring-degree > 1` for a model that only has single-node Ulysses may either raise or, in some cases, silently compute incorrect output; check the model's cookbook page before assuming cross-node scaling is supported. ### Numerics across node boundaries Ring's online-softmax merge across P2P hops accumulates floating-point operations in a different order than single-node attention, so a cross-node run is **not** expected to bit-match a single-node run of the same prompt and seed — this is the same class of difference as choosing a different attention backend, not a correctness regression. What *is* expected: the same request run twice against the same cross-node deployment must produce byte-identical output. Use that repeat-request check, not a cross-topology comparison, to validate a cross-node deployment's determinism. ## Benchmarking Guidance When benchmarking SP, compare the same model, precision, resolution, frame count, step count, scheduler settings, prompt type, and output path. Report both stage latency and peak GPU memory; SP can reduce per-GPU memory while adding communication overhead. Useful metrics: * End-to-end latency * Denoising stage latency * Decoding stage latency * Peak GPU memory and peak allocated memory * Communication or runtime overhead when available ## Reference Benchmark The following numbers are a reference measurement for one setup. They are not a general promise for all Wan2.2 deployments. * Model: `Wan-AI/Wan2.2-TI2V-5B-Diffusers` * Hardware: two 48 GB RTX 40-series GPUs for sequence parallelism, one 48 GB RTX 40-series GPU for baseline * Sequence parallel config: `sp=2, ulysses=1, ring=2` (`u1r2`) * Baseline config: `sp=1, ulysses=1, ring=1` (`u1r1`) ### Stage Time Breakdown
Stage / Metric u1r2 (s) u1r1 baseline (s) Speedup
InputValidation 0.1060 0.1029 0.97x
TextEncoding 1.3965 2.2261 1.59x
LatentPreparation 0.0002 0.0002 1.00x
TimestepPreparation 0.0003 0.0004 1.33x
Denoising 52.6358 71.6785 1.36x
Decoding 7.6708 13.4314 1.75x
Total 63.74 90.63 1.42x
### Memory Usage
Memory Metric u1r2 (GB) u1r1 baseline (GB) Delta
Peak GPU Memory 20.07 27.40 -7.33
Peak Allocated 13.35 20.40 -7.05
Memory Overhead 6.72 7.00 -0.28
Overhead Ratio 33.5% 25.6% +7.9pp
In this setup, end-to-end latency improved from `90.63s` to `63.74s` (`1.42x`) and peak GPU memory dropped by `7.33GB`. The overhead ratio increased, so future tuning should still check communication and runtime overhead on the target hardware. ## Cross-Node Reference Benchmark The following numbers are a reference measurement for MiniMax-H3's cross-node Ulysses × Ring deployment. They are not a general promise for every model or topology — see each model's cookbook page for its own verified cross-node status. * Model: `MiniMaxAI/MiniMax-H3` * Hardware: 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand between nodes * Cross-node config: `--num-gpus 16 --sp-degree 16 --ulysses-degree 8 --ring-degree 2` * Single-node baseline: `--num-gpus 8 --sp-degree 8 --ulysses-degree 8 --ring-degree 1` Denoise-stage-only comparison, holding prompt, seed, and step count fixed:
Task Single-node (s/step) Cross-node (s/step) Change
T2VA denoise 0.749 0.477 -36.3%
Ref2VA / V2V denoise 2.572 1.494 -41.9%
The gain grows with sequence length: ring's per-hop communication cost stays roughly constant while attention compute grows quadratically with sequence length, so V2V's longer packed sequence benefits more than T2VA's shorter one. With the point-to-point KV rotation pipelined against attention compute (rather than a blocking `all_gather`), one V2V request's full denoise stage completed in 68.1-68.3s versus 128.6s on the single-node 8-GPU baseline (-47.0%), with byte-identical output to the unpipelined cross-node path. # Spectrum Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/spectrum Approximate request-scoped denoising-step acceleration. Spectrum forecasts DiT features and skips selected denoising steps. It is an approximation: validate visual or video quality and latency on the exact model, shape, hardware, and sampling settings you plan to deploy. ## Quick start ```bash theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --prompt "A paper boat floating through a misty mountain lake" \ --enable-spectrum \ --save-output ``` ## Scope and constraints * Available only on native FLUX.1, Wan, HunyuanVideo, and SD3 implementation paths. It is not a `--backend diffusers` feature and does not currently cover FLUX.2. * The request control is available through `sglang generate` and Python sampling parameters. It is not a `sglang serve` or OpenAI-server request option yet. * Spectrum and `--enable-teacache` are mutually exclusive. * Start with the defaults. `--debug` adds shadow-prediction validation work, so its latency is not representative of normal Spectrum execution. ## Advanced controls Use `--enable-spectrum` explicitly. Providing any Spectrum override also enables it, but that implicit behavior is intended for scripts rather than new commands. | Flag | Default | Purpose | | -------------------------- | ------- | ---------------------------------------- | | `--spectrum-window-size` | `2.0` | Initial step-skipping window | | `--spectrum-flex-window` | `0.75` | Window growth after a real forward | | `--spectrum-warmup-steps` | `5` | Initial exact DiT forwards | | `--spectrum-m` | `4` | Chebyshev basis count | | `--spectrum-lam` | `0.1` | Ridge regularization | | `--spectrum-tau-num-steps` | `50` | Chebyshev time horizon | | `--history-size` | `100` | Recent feature-history capacity | | `--taylor-order` | `1` | Local predictor order (`1`, `2`, or `3`) | | `--w` | `1.0` | Chebyshev/Taylor blend weight | These controls trade speed against output fidelity. Change one control at a time and retain a lossless baseline for comparison. # Support New Diffusion Models Source: https://docs.sglang.io/docs/sglang-diffusion/support_new_models A concise implementation guide for adding diffusion model families to SGLang-Diffusion. Use this guide as a triage flow for finding the smallest change that can support a model. Most new model work should touch a small set of files, even though the runtime is split into separate folders. ## Read the Code in This Order The files are split by runtime responsibility. For a new model, read the request path first: 1. `registry.py` chooses the model family, sampling params, and pipeline config. 2. `configs/pipeline_configs/{model}.py` defines model-specific denoising and decoding behavior. 3. `runtime/pipelines/{model}.py` wires modules into stages. 4. `runtime/pipelines_core/stages/` runs the shared stage logic. 5. `runtime/models/` contains native model components only when the architecture cannot be reused. That is the dependency direction. Avoid making a model PR that requires readers to jump between folders in a different order. `runtime/models/` owns modeling code: checkpoint-defined neural modules, architecture wrappers, and weight-loading or forward-path details that are intrinsic to one model family. Reusable serving infrastructure belongs in SGLang-Diffusion runtime folders such as `runtime/cache/`, `runtime/distributed/`, `runtime/utils/`, or shared pipeline stages. This includes cache managers, graph runners, process-group transport, request utilities, and common action-policy helpers. Model packages may call these helpers. Keep ownership in shared runtime folders unless the code is truly architecture-specific. ## Out-of-Tree Models and Pipelines An installed package can register native component models and a pipeline without modifying SGLang-Diffusion. Register them in the package's `__init__.py`: ```python theme={null} from sglang.multimodal_gen.registry import register_pipeline from sglang.multimodal_gen.runtime.models.registry import ModelRegistry from .configs import CustomPipelineConfig, CustomSamplingParams from .pipeline import CustomPipeline ModelRegistry.register_model( "CustomTransformer2DModel", "custom_diffusion.models:CustomTransformer2DModel", ) register_pipeline( CustomPipeline, sampling_param_cls=CustomSamplingParams, pipeline_config_cls=CustomPipelineConfig, hf_model_paths=["my-org/custom-diffusion-model"], model_detectors=[lambda value: "custom-diffusion" in value.lower()], ) ``` Install the package in the server environment and set the same environment variable used by SRT plugins: ```bash theme={null} pip install -e /path/to/custom-diffusion SGLANG_EXTERNAL_MODEL_PACKAGE=custom_diffusion \ sglang serve --model-path my-org/custom-diffusion-model ``` Notes: * The string form of `register_model` keeps component imports lazy. * `hf_model_paths` also supports checkpoints without `model_index.json`. Other Diffusers checkpoints can select the pipeline through `_class_name`. * For a standalone safetensors file, pass `--pipeline CustomPipeline`. * Set the environment variable before startup. Each process imports the package once. Use `overwrite=True` only to intentionally replace a built-in pipeline. ## Start With the Smallest Change Before adding files, decide which path fits the model. | Situation | What to do | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | A new checkpoint uses an existing native family | Add the Hugging Face path and, if needed, a small `SamplingParams` or `PipelineConfig` variant. Reuse the existing pipeline and modules. | | The model has a new native DiT/UNet architecture | Add a native SGLang pipeline and the missing model components. Keep denoising and decoding on the shared stages unless measured behavior requires model-specific logic. | | The model is long-tail or you only need compatibility first | Prefer the Diffusers backend for compatibility-first support. Add native support later if performance or deployment needs justify it. | Do not add a folder just to mirror the Diffusers repository layout. Add a new file only when an existing pipeline, stage, module, config, or sampler cannot express the behavior clearly. ## Minimal File Map The source tree is split by runtime responsibility. That split is useful for optimization. Keep new model PRs focused on the files required by model behavior. | Area | Add or edit when | Typical file | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | Registry | Always, unless extending an already registered family | `python/sglang/multimodal_gen/registry.py` | | Runtime parameters | The request schema differs from existing models | `configs/sample/{model}.py` | | Pipeline config | Denoising, decoding, precision, position encoding, or CFG hooks differ | `configs/pipeline_configs/{model}.py` | | Pipeline wiring | The model needs a new stage layout or module list | `runtime/pipelines/{model}.py` | | DiT/UNet module | The denoising network is new | `runtime/models/dits/{model}.py` | | dVLA or policy module | The checkpoint defines a new action-policy architecture | `runtime/models/{family}/modeling_*.py` or a task-specific model subfolder | | Shared runtime infrastructure | Cache, CUDA graph, distributed transfer, request utilities, or action-policy helpers can be reused by future models | `runtime/cache/`, `runtime/distributed/`, `runtime/utils/`, `runtime/pipelines_core/stages/` | | Model component config | A model component has static architecture config | `configs/models/dits/{model}.py`, `configs/models/vaes/{model}.py` | | Model-specific stage | A single stage's runtime semantics cannot be expressed by a native stage or a narrow subclass of one | `runtime/pipelines_core/stages/model_specific_stages/{model}.py` | | Encoder, VAE, scheduler | No existing implementation can be reused | `runtime/models/encoders/`, `runtime/models/vaes/`, `runtime/models/schedulers/` | For a new native architecture, the common minimum is: 1. `registry.py` 2. `configs/sample/{model}.py` 3. `configs/pipeline_configs/{model}.py` 4. `runtime/pipelines/{model}.py` 5. `runtime/models/dits/{model}.py` Every extra file should map to model behavior that existing code cannot express clearly. For dVLA or other non-image diffusion policies, keep the same ownership rule. The policy network, VLM/action expert modules, checkpoint mapping, and model-specific forward code belong under `runtime/models/`. Prefix caches, request-local contexts, denoising graph runners, OpenPI-compatible transport, and prefix/action process-group utilities should be shared SGLang-Diffusion runtime infrastructure when they are useful beyond the first model. ## Read the Reference First Use the model's Diffusers pipeline, official implementation, or `model_index.json` as the source of truth. Write down: * Which modules must be loaded: tokenizer, text encoder, image encoder, transformer, scheduler, VAE, processor, and any extra adapters. * The prompt and image encoding flow. * Latent shape, packing, scale, shift, dtype, and device rules. * Timestep and sigma schedule. * The exact `forward()` kwargs expected by the denoising network. * VAE decode rules and output post-processing. If the new model is close to Flux, Qwen-Image, GLM-Image, Wan, HunyuanVideo, or LTX, extend that implementation before starting from an empty file. ## Choose a Pipeline Shape SGLang-Diffusion uses `ComposedPipelineBase` to wire stages together. Most native pipelines should choose the least invasive stage shape that preserves the runtime semantics. | Shape | Use when | Layout | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Native stages | Text/image encoding, latent prep, timestep prep, denoising, and decoding match existing helpers | `add_standard_t2i_stages()`, `add_standard_ti2i_stages()`, or a similar helper | | Native-stage subclass | One stage has model-specific details, but the stage boundary and batch contract still match an existing native stage | `{Model}TextEncodingStage(TextEncodingStage) -> LatentPreparationStage -> TimestepPreparationStage -> DenoisingStage -> DecodingStage` | | Custom single-purpose stage | One step has a different state owner or batch-field lifecycle and cannot cleanly inherit from a native stage | Native stages with one `{Model}{Purpose}Stage` inserted or substituted | | Aggregated custom stage | Several preparation steps are inseparable in the reference pipeline and cannot be split without fragile duplicate state | `{Model}BeforeDenoisingStage -> DenoisingStage -> DecodingStage` | Prefer this order: 1. **Use native stages directly.** This keeps the model on shared code paths for offload, component readiness, profiling, disaggregation, batching, and future stage-level optimizations. 2. **Subclass the narrowest native stage.** If only prompt processing differs, inherit from `TextEncodingStage`. If only latent setup, timestep setup, denoising, or decode differs, inherit from that specific native stage. Preserve the existing input/output fields whenever possible. 3. **Add a custom single-purpose stage only when no native stage contract fits.** Keep the stage owner narrow: one stage should own one coherent transformation, such as a custom condition assembly step or a model-specific policy/action bridge. 4. **Use an aggregated `BeforeDenoisingStage` only as a last resort.** This is the least preferred shape because it hides multiple runtime responsibilities in one stage, increases code size and review cost, and bypasses shared hooks for offload, profiling, disaggregation, batching, and future stage-level optimizations. ## Implement the Pieces ### 1. Sampling Params Create request parameters only for values users can set at runtime. ```python theme={null} # python/sglang/multimodal_gen/configs/sample/my_model.py from dataclasses import dataclass from sglang.multimodal_gen.configs.sample.sampling_params import ImageSamplingParams @dataclass class MyModelSamplingParams(ImageSamplingParams): guidance_scale: float = 4.0 num_inference_steps: int = 28 ``` ### 2. Pipeline Config `PipelineConfig` is where shared denoising and decoding stages get model-specific callbacks. ```python theme={null} # python/sglang/multimodal_gen/configs/pipeline_configs/my_model.py from dataclasses import dataclass, field @dataclass class MyModelPipelineConfig(ImagePipelineConfig): task_type: ModelTaskType = ModelTaskType.T2I should_use_guidance: bool = True dit_config: DiTConfig = field(default_factory=MyModelDiTConfig) vae_config: VAEConfig = field(default_factory=MyModelVAEConfig) def prepare_pos_cond_kwargs(self, batch, latent_model_input, t, **kwargs): return { "hidden_states": latent_model_input, "encoder_hidden_states": batch.prompt_embeds[0], "timestep": t, } def prepare_neg_cond_kwargs(self, batch, latent_model_input, t, **kwargs): return { "hidden_states": latent_model_input, "encoder_hidden_states": batch.negative_prompt_embeds[0], "timestep": t, } ``` Make these kwargs match the denoising module's `forward()` signature exactly. ### 3. Pipeline Wiring Use the standard helper when the model fits it. ```python theme={null} # python/sglang/multimodal_gen/runtime/pipelines/my_model.py class MyModelPipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "MyModelPipeline" _required_config_modules = [ "text_encoder", "tokenizer", "transformer", "scheduler", "vae", ] def create_pipeline_stages(self, server_args: ServerArgs): self.add_standard_t2i_stages() EntryClass = [MyModelPipeline] ``` When a standard helper is not enough, first check whether only one native stage needs model-specific behavior. In that case, subclass that stage and keep the rest of the pipeline standard. For example, custom tokenization or prompt-window logic should usually inherit from `TextEncodingStage` directly. ```python theme={null} class MyModelPipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "MyModelPipeline" _required_config_modules = [ "text_encoder", "tokenizer", "transformer", "scheduler", "vae", ] def create_pipeline_stages(self, server_args: ServerArgs): self.add_stage(InputValidationStage()) self.add_stage_factory( RoleType.ENCODER, lambda: MyModelTextEncodingStage( text_encoder=self.get_module("text_encoder"), tokenizer=self.get_module("tokenizer"), ), "my_model_text_encoding_stage", ) self.add_standard_latent_preparation_stage() self.add_standard_timestep_preparation_stage() self.add_standard_denoising_stage() self.add_standard_decoding_stage() EntryClass = [MyModelPipeline] ``` Use a custom single-purpose stage only when the reference pipeline has one step that cannot be represented cleanly by a hook or native-stage subclass. Keep the custom stage narrow and reuse native stages before and after it. ```python theme={null} class MyModelPipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "MyModelPipeline" _required_config_modules = [ "text_encoder", "tokenizer", "transformer", "scheduler", "vae", ] def create_pipeline_stages(self, server_args: ServerArgs): self.add_stage(InputValidationStage()) self.add_standard_text_encoding_stage() self.add_stage_factory( RoleType.ENCODER, lambda: MyModelConditioningStage( scheduler=self.get_module("scheduler"), vae=self.get_module("vae"), ), "my_model_conditioning_stage", ) self.add_standard_latent_preparation_stage() self.add_standard_timestep_preparation_stage() self.add_standard_denoising_stage() self.add_standard_decoding_stage() EntryClass = [MyModelPipeline] ``` Use an aggregated `BeforeDenoisingStage` only when the reference pipeline couples several preparation steps so tightly that splitting them would require fragile duplicate state or extra synchronization. Do not start with this shape. ```python theme={null} class MyModelPipeline(LoRAPipeline, ComposedPipelineBase): pipeline_name = "MyModelPipeline" _required_config_modules = [ "text_encoder", "tokenizer", "transformer", "scheduler", "vae", ] def create_pipeline_stages(self, server_args: ServerArgs): self.add_stage(InputValidationStage()) self.add_stage( MyModelBeforeDenoisingStage( text_encoder=self.get_module("text_encoder"), tokenizer=self.get_module("tokenizer"), scheduler=self.get_module("scheduler"), vae=self.get_module("vae"), ) ) self.add_standard_denoising_stage() self.add_standard_decoding_stage() EntryClass = [MyModelPipeline] ``` ### 4. Last-Resort Before-Denoising Stage A `BeforeDenoisingStage` is not a catch-all replacement for the native stages. Use it when the model has custom latent packing, conditioning assembly, timestep preparation, or request-local state that does not fit `LatentPreparationStage` or `TimestepPreparationStage`, and only after checking whether the work can be a native-stage subclass or a custom single-purpose stage. If the difference is prompt handling, subclass `TextEncodingStage` instead. A proper `BeforeDenoisingStage` should populate the batch fields consumed by `DenoisingStage`. ```python theme={null} class MyModelBeforeDenoisingStage(PipelineStage): @torch.no_grad() def forward(self, batch: Req, server_args: ServerArgs) -> Req: prompt_embeds, negative_prompt_embeds = self._encode_prompt(batch) latents = self._prepare_latents(batch) timesteps, sigmas = self._prepare_timesteps(batch) batch.prompt_embeds = [prompt_embeds] batch.negative_prompt_embeds = [negative_prompt_embeds] batch.latents = latents batch.timesteps = timesteps batch.num_inference_steps = len(timesteps) batch.sigmas = sigmas.tolist() batch.raw_latent_shape = latents.shape return batch ``` Required fields for `DenoisingStage`: | Field | Notes | | ------------------------------ | ------------------------------------------------------------------- | | `batch.latents` | Initial latent tensor, including any packing required by the model. | | `batch.timesteps` | Timestep tensor in the exact order used by the reference pipeline. | | `batch.sigmas` | Python list when the scheduler expects sigma values. | | `batch.prompt_embeds` | Positive embeddings, wrapped in a list. | | `batch.negative_prompt_embeds` | Negative embeddings, wrapped in a list when CFG is used. | | `batch.num_inference_steps` | Number of denoising iterations. | | `batch.raw_latent_shape` | Original latent shape before packing, if decode needs it. | ### 5. Distributed and memory integration Single-GPU parity is only the first milestone. Complete native support also requires: * **Encoder and DiT TP/SP:** use native parallel projections and sharded weight loading for TP, and `USPAttention` for SP. Handle masks, RoPE, padding, and output gathering without falling back to a replicated full model. TP and SP must work together. * **VAE parallel decode:** subclass `ParallelTiledVAE`, or reuse an existing native base with the same contract. Support tiled and `spatial_shard` decode through `DecodingStage` and the shared decode group. Reuse `runtime/layers/parallel_conv.py` and `runtime/models/vaes/parallel/diffusers_spatial.py` where applicable. * **Layerwise offload:** every loaded neural module must inherit `LayerwiseOffloadableModuleMixin` and list all repeated block paths in `layer_names`. Set `layerwise_offload_dit_group_enabled = False` for non-DiT modules. Component CPU offload is not a substitute. See `wanvideo.py` and `qwen_image.py` for DiT TP/SP, `gemma_3.py` for encoder TP and offload, and `autoencoder_kl_qwenimage.py` or `ltx_2_vae.py` for VAE decode. The Diffusers backend is compatibility-first and does not need to meet this native integration contract. ### 6. Registry Register the family once the sampling params and pipeline config exist. ```python theme={null} register_configs( model_family="my_model", sampling_param_cls=MyModelSamplingParams, pipeline_config_cls=MyModelPipelineConfig, hf_model_paths=["org/my-model"], ) ``` The pipeline file is discovered through its `EntryClass`; do not add a second pipeline registry unless the existing registry requires it. ## Verify the Port Use one deterministic prompt and seed while comparing with the reference implementation. 1. Run a single-GPU smoke test and check that the output contains coherent content. 2. Compare latent scale and shift, timestep order, sigma values, and conditioning kwargs against Diffusers or the official implementation. 3. Compare VAE decode separately, including tiled and multi-GPU `spatial_shard`. 4. Run encoder and DiT TP, SP, combined TP x SP, and `--layerwise-offload-components all`; compare with the single-GPU resident baseline. 5. If the model supports LoRA, CFG parallelism, or disaggregation, test each feature explicitly. 6. Add or update docs, examples, or the compatibility matrix when users need a new launch command. Common failure points: * Wrong latent scale or shift. * Reversed or dtype-mismatched timesteps. * Missing negative embeddings when CFG is enabled. * Conditioning kwarg names mismatched with the DiT `forward()`. * Rotary embedding shape or style mismatch. * Decoding packed latents without restoring `raw_latent_shape`. ## PR Checklist * [ ] Reused an existing family, stage, module, scheduler, or VAE wherever possible. * [ ] Kept the new-model touch surface small and justified any extra files. * [ ] Added `SamplingParams`, `PipelineConfig`, pipeline wiring, DiT module, and registry entry when native support is needed. * [ ] Confirmed `pipeline_name` matches the Diffusers `model_index.json` `_class_name` when applicable. * [ ] Confirmed `_required_config_modules` matches the model repo. * [ ] Verified image or video quality against a reference output. * [ ] Completed the distributed and memory integration checks above. * [ ] Tested CFG parallelism and distributed serving paths when they apply. # TeaCache Acceleration Source: https://docs.sglang.io/docs/sglang-diffusion/teacache Configure TeaCache for temporal similarity-based diffusion acceleration. > **Note**: This is one of two caching strategies available in SGLang. > For an overview of all caching options, see [caching](./caching-acceleration). TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely. ## Overview TeaCache works by: 1. Tracking the L1 distance between modulated inputs across consecutive timesteps 2. Accumulating the rescaled L1 distance over steps 3. When accumulated distance is below a threshold, reusing the cached residual 4. Using separate positive/negative caches for supported CFG model families ## How It Works ### L1 Distance Tracking At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs: ```text theme={null} rel_l1 = |current - previous|.mean() / |previous|.mean() ``` This distance is then rescaled using polynomial coefficients and accumulated: ```text theme={null} accumulated += poly(coefficients)(rel_l1) ``` ### Cache Decision * If `accumulated >= threshold`: Force computation, reset accumulator * If `accumulated < threshold`: Skip computation, use cached residual ### CFG Support For models that support CFG cache separation, TeaCache maintains separate caches for positive and negative branches: * `previous_modulated_input` / `previous_residual` for positive branch * `previous_modulated_input_negative` / `previous_residual_negative` for negative branch For models that do not support CFG separation, TeaCache is automatically disabled when CFG is enabled. ## Configuration TeaCache is configured via `TeaCacheParams` in the sampling parameters: ```python theme={null} from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams params = TeaCacheParams( teacache_thresh=0.1, # Threshold for accumulated L1 distance coefficients=[1.0, 0.0, 0.0], # Polynomial coefficients for L1 rescaling ) ``` ### Parameters
Parameter Type Description
`teacache_thresh` float Threshold for accumulated L1 distance. Higher = more caching, faster but potentially lower quality
`coefficients` list\[float] Polynomial coefficients for L1 rescaling. Model-specific tuning
### Model-Specific Configurations Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality. ## Supported Models TeaCache support status by model family:
Model Family CFG Cache Separation Notes
Wan2.1 Yes Full support
Wan2.2 Yes Coefficients are not calibrated yet; enabling TeaCache is accepted but currently no-ops
Z-Image Yes Full support
HunyuanVideo No Not supported yet
Flux No To be supported
Qwen No To be supported
## References * [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324) # Supported models Source: https://docs.sglang.io/docs/supported-models See which families of SGLang-compatible models are actively maintained. SGLang supports model families across text generation, retrieval, and reward workflows. Browse the sections below for the primary product paths and jump to the detail pages when you are ready to explore a specific class. ### Text generation Production-tuned Llama and Qwen families validated for high-throughput serving. Vision-text hybrids that stay responsive on multi-GPU setups. Score-based and diffusion backbones for structured text generation workflows. ### Retrieval and ranking Dense and sparse embeddings optimized with FlashInfer kernels. Low-latency rerankers for multi-stage retrieval pipelines. Lightweight classifiers covering safety, intent, and context filters. ### Specialized models RLHF and reward scoring pipelines optimized for production latency. # Classification Models Source: https://docs.sglang.io/docs/supported-models/classify_models This document describes the `/v1/classify` API endpoint implementation in SGLang, which is compatible with vLLM's classification API format. ## Overview The classification API allows you to classify text inputs using classification models. This implementation follows the same format as vLLM's 0.7.0 classification API. ## API Endpoint ```text Output theme={null} POST /v1/classify ``` ## Request Format ```json Config theme={null} { "model": "model_name", "input": "text to classify" } ``` ### Parameters * `model` (string, required): The name of the classification model to use * `input` (string, required): The text to classify * `user` (string, optional): User identifier for tracking * `rid` (string, optional): Request ID for tracking * `priority` (integer, optional): Request priority ## Response Format ```json Config theme={null} { "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682", "object": "list", "created": 1745383213, "model": "jason9693/Qwen2.5-1.5B-apeach", "data": [ { "index": 0, "label": "Default", "probs": [0.565970778465271, 0.4340292513370514], "num_classes": 2 } ], "usage": { "prompt_tokens": 10, "total_tokens": 10, "completion_tokens": 0, "prompt_tokens_details": null } } ``` ### Response Fields * `id`: Unique identifier for the classification request * `object`: Always "list" * `created`: Unix timestamp when the request was created * `model`: The model used for classification * `data`: Array of classification results * `index`: Index of the result * `label`: Predicted class label * `probs`: Array of probabilities for each class * `num_classes`: Total number of classes * `usage`: Token usage information * `prompt_tokens`: Number of input tokens * `total_tokens`: Total number of tokens * `completion_tokens`: Number of completion tokens (always 0 for classification) * `prompt_tokens_details`: Additional token details (optional) ## Example Usage ### Using curl ```bash Command theme={null} curl -v "http://127.0.0.1:8000/v1/classify" \ -H "Content-Type: application/json" \ -d '{ "model": "jason9693/Qwen2.5-1.5B-apeach", "input": "Loved the new café—coffee was great." }' ``` ### Using Python ```python Example theme={null} import requests import json # Make classification request response = requests.post( "http://127.0.0.1:8000/v1/classify", headers={"Content-Type": "application/json"}, json={ "model": "jason9693/Qwen2.5-1.5B-apeach", "input": "Loved the new café—coffee was great." } ) # Parse response result = response.json() print(json.dumps(result, indent=2)) ``` ## Supported Models The classification API works with any classification model supported by SGLang, including: ### Classification Models (Multi-class) * `LlamaForSequenceClassification` - Multi-class classification * `Qwen2ForSequenceClassification` - Multi-class classification * `Qwen3ForSequenceClassification` - Multi-class classification * `BertForSequenceClassification` - Multi-class classification * `Gemma2ForSequenceClassification` - Multi-class classification **Label Mapping**: The API automatically uses the `id2label` mapping from the model's `config.json` file to provide meaningful label names instead of generic class names. If `id2label` is not available, it falls back to `LABEL_0`, `LABEL_1`, etc., or `Class_0`, `Class_1` as a last resort. ### Reward Models (Single score) * `InternLM2ForRewardModel` - Single reward score * `Qwen2ForRewardModel` - Single reward score * `LlamaForSequenceClassificationWithNormal_Weights` - Special reward model **Note**: The `/classify` endpoint in SGLang was originally designed for reward models but now supports all non-generative models. Our `/v1/classify` endpoint provides a standardized vLLM-compatible interface for classification tasks. ## Error Handling The API returns appropriate HTTP status codes and error messages: * `400 Bad Request`: Invalid request format or missing required fields * `500 Internal Server Error`: Server-side processing error Error response format: ```json Config theme={null} { "error": "Error message", "type": "error_type", "code": 400 } ``` ## Implementation Details The classification API is implemented using: 1. **Rust Model Gateway**: Handles routing and request/response models in `sgl-model-gateway/src/protocols/spec.rs` 2. **Python HTTP Server**: Implements the actual endpoint in `python/sglang/srt/entrypoints/http_server.py` 3. **Classification Service**: Handles the classification logic in `python/sglang/srt/entrypoints/openai/serving_classify.py` ## Testing Use the provided test script to verify the implementation: ```bash Command theme={null} python test_classify_api.py ``` ## Compatibility This implementation is compatible with vLLM's classification API format, allowing seamless migration from vLLM to SGLang for classification tasks. # Diffusion language models Source: https://docs.sglang.io/docs/supported-models/diffusion_language_models Diffusion language models have shown promise for non-autoregressive text generation with parallel decoding capabilities. Unlike auto-regressive language models, different diffusion language models require different decoding strategies. ## Example Launch Command SGLang supports different DLLM algorithms such as `LowConfidence` and `JointThreshold`. ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.0-mini \ # example HF/local path --dllm-algorithm LowConfidence \ --dllm-algorithm-config ./config.yaml \ # Optional. Uses the algorithm's default if not set. --host 0.0.0.0 \ --port 30000 ``` ## First-Done-First-Out (FDFO) Scheduling FDFO scheduling is **enabled by default**: each request leaves the batch as soon as its block is resolved, instead of advancing in lockstep where fast-converging requests must wait for slow long-tail requests before leaving the batch (head-of-line blocking). This improves throughput and is orthogonal to `--dllm-algorithm`, so it works with any dLLM algorithm. Pass `--no-dllm-fdfo` to fall back to synchronous lockstep scheduling: ```bash Command theme={null} python3 -m sglang.launch_server \ --model-path inclusionAI/LLaDA2.0-mini \ --dllm-algorithm LowConfidence \ --no-dllm-fdfo \ --host 0.0.0.0 \ --port 30000 ``` ## Example Configuration File Depending on the algorithm selected, the configuration parameters vary. LowConfidence Config: ```yaml Config theme={null} # Confidence threshold for accepting predicted tokens # - Higher values: More conservative, better quality but slower # - Lower values: More aggressive, faster but potentially lower quality # Range: 0.0 - 1.0 threshold: 0.95 # Default: 32, for LLaDA2MoeModelLM block_size: 32 ``` JointThreshold Config: ```yaml Config theme={null} # Decoding threshold for Mask-to-Token (M2T) phase # - Higher values: More conservative, better quality but slower # - Lower values: More aggressive, faster but potentially lower quality # Range: 0.0 - 1.0 threshold: 0.5 # Decoding threshold for Token-to-Token (T2T) phase # Range: 0.0 - 1.0 # Setting to 0.0 allows full editing (recommended for most cases). edit_threshold: 0.0 # Max extra T2T steps after all masks are removed. Prevents infinite loops. max_post_edit_steps: 16 # 2-gram repetition penalty (default 0). # An empirical value of 3 is often sufficient to mitigate most repetitions. penalty_lambda: 0 ``` ## Example Client Code Snippet Just like other supported models, diffusion language models can be used via the REST API or Python client. Python client example for making a generation request to the launched server: ```python Example theme={null} import sglang as sgl def main(): llm = sgl.Engine(model_path="inclusionAI/LLaDA2.0-mini", dllm_algorithm="LowConfidence", max_running_requests=1, trust_remote_code=True) prompts = [ "SYSTEMdetailed thinking off<|role_end|>HUMAN Write a brief introduction of the great wall <|role_end|>ASSISTANT" ] sampling_params = { "temperature": 0, "max_new_tokens": 1024, } outputs = llm.generate(prompts, sampling_params) print(outputs) if __name__ == '__main__': main() ``` Curl example for making a generation request to the launched server: ```bash Command theme={null} curl -X POST "http://127.0.0.1:30000/generate" \ -H "Content-Type: application/json" \ -d '{ "text": [ "SYSTEMdetailed thinking off<|role_end|>HUMAN Write the number from 1 to 128 <|role_end|>ASSISTANT", "SYSTEMdetailed thinking off<|role_end|>HUMAN Write a brief introduction of the great wall <|role_end|>ASSISTANT" ], "stream": true, "sampling_params": { "temperature": 0, "max_new_tokens": 1024 } }' ``` ## Supported Models Below the supported models are summarized in a table.
Model Family Example Model Description
LLaDA2.0 (mini, flash) inclusionAI/LLaDA2.0-flash LLaDA2.0-flash is a diffusion language model featuring a 100B Mixture-of-Experts (MoE) architecture.
SDAR (JetLM) JetLM/SDAR-8B-Chat SDAR series diffusion language model (Chat), dense architecture.
SDAR (JetLM) JetLM/SDAR-30B-A3B-Chat SDAR series diffusion language model (Chat), MoE architecture.
DiffusionGemma google/diffusiongemma-26B-A4B-it Uniform-state (renoising) block-diffusion multimodal (text + image) MoE, 25.2B total / 3.8B active, served with the Gemma4Renoise sampler.
# Embedding models Source: https://docs.sglang.io/docs/supported-models/embedding_models Dense and sparse embedding models with FlashInfer acceleration and SGLang's batching infrastructure. SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it. ## Quick Start ### Launch Server ```bash theme={null} sglang serve \ --model-path Qwen/Qwen3-Embedding-4B \ --is-embedding ``` ### EmbeddingGemma EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply: ```bash theme={null} sglang serve --model-path google/embeddinggemma-300m ``` On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs. ### Client Request ```python theme={null} import requests url = "http://127.0.0.1:30000" payload = { "model": "Qwen/Qwen3-Embedding-4B", "input": "What is the capital of France?", "encoding_format": "float" # or "base64" for compact FP32 responses } response = requests.post(url + "/v1/embeddings", json=payload).json() print("Embedding:", response["data"][0]["embedding"]) ``` ## Multimodal Embedding Example For multimodal models like GME that support both text and images: ```bash theme={null} sglang serve \ --model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \ --is-embedding \ --chat-template gme-qwen2-vl ``` ```python Example theme={null} import requests url = "http://127.0.0.1:30000" text_input = "Represent this image in embedding space." image_path = "https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild/resolve/main/images/023.jpg" payload = { "model": "gme-qwen2-vl", "input": [ { "text": text_input }, { "image": image_path } ], } response = requests.post(url + "/v1/embeddings", json=payload).json() print("Embeddings:", [x.get("embedding") for x in response.get("data", [])]) ``` ## Matryoshka Embedding Example [Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost. ### 1. Launch a Matryoshka‑capable model If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below: ```bash Command theme={null} sglang serve \ --model-path Qwen/Qwen3-Embedding-0.6B \ --is-embedding \ --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}' ``` 1. Setting `"is_matryoshka": true` allows truncating to any dimension. Otherwise, the server will validate that the specified dimension in the request is one of `matryoshka_dimensions`. 2. Omitting `dimensions` in a request returns the full vector. ### 2. Make requests with different output dimensions ```python theme={null} import requests url = "http://127.0.0.1:30000" # Request a truncated (Matryoshka) embedding by specifying a supported dimension. payload = { "model": "Qwen/Qwen3-Embedding-0.6B", "input": "Explain diffusion models simply.", "dimensions": 512 # change to 128 / 1024 / omit for full size } response = requests.post(url + "/v1/embeddings", json=payload).json() print("Embedding:", response["data"][0]["embedding"]) ``` ## Supported Models
Model Family Example Model Chat template Description
EmbeddingGemma `google/embeddinggemma-300m` N/A Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default
E5 (Llama/Mistral based) `intfloat/e5-mistral-7b-instruct` N/A High-quality text embeddings based on Mistral/Llama architectures
GTE-Qwen2 `Alibaba-NLP/gte-Qwen2-7B-instruct` N/A Alibaba's text embedding model with multilingual support
Qwen3-Embedding `Qwen/Qwen3-Embedding-4B` N/A Latest Qwen3-based text embedding model for semantic representation
Qwen3 (bare backbone) `microsoft/harrier-oss-v1-0.6b` N/A Bare Qwen3Model backbone (no LM head); served natively on SGLang's fused Qwen3 kernels and auto-classified as an embedding model
BGE `BAAI/bge-large-en-v1.5` N/A BAAI's text embeddings (requires attention-backend triton/torch\_native)
GME (Multimodal) `Alibaba-NLP/gme-Qwen2-VL-2B-Instruct` `gme-qwen2-vl` Multimodal embedding for text and image cross-modal tasks
CLIP `openai/clip-vit-large-patch14-336` N/A OpenAI's CLIP for image and text embeddings
# Large Language Models Source: https://docs.sglang.io/docs/supported-models/generative_models These models accept text input and produce text output (e.g., chat completions). They are primarily large language models (LLMs), some with mixture-of-experts (MoE) architectures for scaling. ## Example launch Command ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.2-1B-Instruct \ # example HF/local path --host 0.0.0.0 \ --port 30000 \ ``` ## Supported models Below the supported models are summarized in a table. If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen3ForCausalLM`, use the expression: ```text Output theme={null} repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen3ForCausalLM ``` in the GitHub search bar.
Model Family (Variants) Example HuggingFace Identifier Description
**DeepSeek** (v1, v2, v3/R1) `deepseek-ai/DeepSeek-R1` Series of advanced reasoning-optimized models (including a 671B MoE) trained with reinforcement learning; top performance on complex reasoning, math, and code tasks. SGLang provides Deepseek v3/R1 model-specific optimizations and Reasoning Parser
**Kimi K2** (Thinking, Instruct) `moonshotai/Kimi-K2-Instruct` Moonshot AI's 1 trillion parameter MoE model (32B active) with 128K–256K context; state-of-the-art agentic intelligence with stable long-horizon agency across 200–300 sequential tool calls. Features MLA attention and native INT4 quantization. See Reasoning Parser docs
**Kimi Linear** (48B-A3B) `moonshotai/Kimi-Linear-48B-A3B-Instruct` Moonshot AI's hybrid linear attention model (48B total, 3B active) with 1M token context; features Kimi Delta Attention (KDA) for up to 6× faster decoding and 75% KV cache reduction vs full attention.
**GPT-OSS** openai/gpt-oss-20b, openai/gpt-oss-120b OpenAI’s latest GPT-OSS series for complex reasoning, agentic tasks, and versatile developer use cases.
Qwen (3.5, 3, 3MoE, 3Next, 2.5, 2 series) Qwen/Qwen3.5-397B-A17B, Qwen/Qwen3-0.6B, Qwen/Qwen3-30B-A3B, Qwen/Qwen3-Next-80B-A3B-Instruct Alibaba’s latest Qwen3 series for complex reasoning, language understanding, and generation tasks; Support for MoE variants along with previous generation 2.5, 2, etc. SGLang provides Qwen3 specific reasoning parser
**Llama** (2, 3.x, 4 series) `meta-llama/Llama-4-Scout-17B-16E-Instruct` Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. SGLang provides Llama-4 model-specific optimizations
**Mistral** (Mixtral, NeMo, Small3) `mistralai/Mistral-7B-Instruct-v0.2` Open 7B LLM by Mistral AI with strong performance; extended into MoE (“Mixtral”) and NeMo Megatron variants for larger scale.
**Gemma** (v1, v2, v3) `google/gemma-3-1b-it` Google’s family of efficient multilingual models (1B–27B); Gemma 3 offers a 128K context window, and its larger (4B+) variants support vision input.
**Phi** (Phi-1.5, Phi-2, Phi-3, Phi-4, Phi-MoE series) microsoft/Phi-4-multimodal-instruct, microsoft/Phi-3.5-MoE-instruct Microsoft’s Phi family of small models (1.3B–5.6B); Phi-4-multimodal (5.6B) processes text, images, and speech, Phi-4-mini is a high-accuracy text model and Phi-3.5-MoE is a mixture-of-experts model.
**MiniCPM** (v3, 4B) `openbmb/MiniCPM3-4B` OpenBMB’s series of compact LLMs for edge devices; MiniCPM 3 (4B) achieves GPT-3.5-level results in text tasks.
**OLMo** (2, 3) allenai/OLMo-3-1125-32B, allenai/OLMo-3-32B-Think, allenai/OLMo-2-1124-7B-Instruct Allen AI’s series of Open Language Models designed to enable the science of language models.
**OLMoE** (Open MoE) `allenai/OLMoE-1B-7B-0924` Allen AI’s open Mixture-of-Experts model (7B total, 1B active parameters) delivering state-of-the-art results with sparse expert activation.
MiniMax-M2 (M2, M2.1, M2.5) MiniMaxAI/MiniMax-M2.5, MiniMaxAI/MiniMax-M2.1, MiniMaxAI/MiniMax-M2 MiniMax's SOTA LLM for coding & agentic workflows.
**StableLM** (3B, 7B) `stabilityai/stablelm-tuned-alpha-7b` StabilityAI’s early open-source LLM (3B & 7B) for general text generation; a demonstration model with basic instruction-following ability.
**Command-(R,A)** (Cohere) CohereLabs/c4ai-command-r-v01, CohereLabs/c4ai-command-r7b-12-2024, CohereLabs/c4ai-command-a-03-2025 Cohere’s open conversational LLM (Command series) optimized for long context, retrieval-augmented generation, and tool use.
**DBRX** (Databricks) `databricks/dbrx-instruct` Databricks’ 132B-parameter MoE model (36B active) trained on 12T tokens; competes with GPT-3.5 quality as a fully open foundation model.
**Grok** (xAI) `xai-org/grok-1` xAI’s grok-1 model known for vast size(314B parameters) and high quality; integrated in SGLang for high-performance inference.
**ChatGLM** (GLM-130B family) `THUDM/chatglm2-6b` Zhipu AI’s bilingual chat model (6B) excelling at Chinese-English dialogue; fine-tuned for conversational quality and alignment.
**InternLM 2** (7B, 20B) `internlm/internlm2-7b` Next-gen InternLM (7B and 20B) from SenseTime, offering strong reasoning and ultra-long context support (up to 200K tokens).
**ExaONE 3** (Korean-English) `LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct` LG AI Research’s Korean-English model (7.8B) trained on 8T tokens; provides high-quality bilingual understanding and generation.
**Baichuan 2** (7B, 13B) `baichuan-inc/Baichuan2-13B-Chat` BaichuanAI’s second-generation Chinese-English LLM (7B/13B) with improved performance and an open commercial license.
**XVERSE** (MoE) `xverse/XVERSE-MoE-A36B` Yuanxiang’s open MoE LLM (XVERSE-MoE-A36B: 255B total, 36B active) supporting \~40 languages; delivers 100B+ dense-level performance via expert routing.
**SmolLM** (135M–1.7B) `HuggingFaceTB/SmolLM-1.7B` Hugging Face’s ultra-small LLM series (135M–1.7B params) offering surprisingly strong results, enabling advanced AI on mobile/edge devices.
**GLM-4** (Multilingual 9B) `ZhipuAI/glm-4-9b-chat` Zhipu’s GLM-4 series (up to 9B parameters) – open multilingual models with support for 1M-token context and even a 5.6B multimodal variant (Phi-4V).
**MiMo** (7B series) `XiaomiMiMo/MiMo-7B-RL` Xiaomi's reasoning-optimized model series, leverages Multiple-Token Prediction for faster inference.
**ERNIE-4.5** (4.5, 4.5MoE series) `baidu/ERNIE-4.5-21B-A3B-PT` Baidu's ERNIE-4.5 series which consists of MoE with 47B and 3B active parameters, with the largest model having 424B total parameters, as well as a 0.3B dense model.
**Arcee AFM-4.5B** `arcee-ai/AFM-4.5B-Base` Arcee's foundational model series for real world reliability and edge deployments.
**Persimmon** (8B) `adept/persimmon-8b-chat` Adept’s open 8B model with a 16K context window and fast inference; trained for broad usability and licensed under Apache 2.0.
**Solar** (10.7B) `upstage/SOLAR-10.7B-Instruct-v1.0` Upstage's 10.7B parameter model, optimized for instruction-following tasks. This architecture incorporates a depth-up scaling methodology, enhancing model performance.
**Tele FLM** (52B-1T) `CofeAI/Tele-FLM` BAAI & TeleAI's multilingual model, available in 52-billion and 1-trillion parameter variants. It is a decoder-only transformer trained on \~2T tokens
**Ling** (16.8B–290B) inclusionAI/Ling-lite, inclusionAI/Ling-plus InclusionAI’s open MoE models. Ling-Lite has 16.8B total / 2.75B active parameters, and Ling-Plus has 290B total / 28.8B active parameters. They are designed for high performance on NLP and complex reasoning tasks.
**Granite 3.0, 3.1** (IBM) `ibm-granite/granite-3.1-8b-instruct` IBM's open dense foundation models optimized for reasoning, code, and business AI use cases. Integrated with Red Hat and watsonx systems.
**Granite 3.0, 3.1 MoE** (IBM) `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm-granite/granite-3.1-3b-a800m-instruct` IBM’s Mixture-of-Experts models offering strong performance with cost-efficiency. MoE expert routing designed for enterprise deployment at scale. 3.1 adds shared experts.
**GPT-J** (6B) `EleutherAI/gpt-j-6b` EleutherAI's GPT-2-like causal language model (6B) trained on the Pile dataset.
**Orion** (14B) `OrionStarAI/Orion-14B-Base` A series of open-source multilingual large language models by OrionStarAI, pretrained on a 2.5T token multilingual corpus including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages.
**Llama Nemotron Super** (v1, v1.5, NVIDIA) nvidia/Llama-3\_3-Nemotron-Super-49B-v1, nvidia/Llama-3\_3-Nemotron-Super-49B-v1\_5 The NVIDIA Nemotron family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents.
**Llama Nemotron Ultra** (v1, NVIDIA) `nvidia/Llama-3_1-Nemotron-Ultra-253B-v1` The NVIDIA Nemotron family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents.
**NVIDIA Nemotron Nano 2.0** `nvidia/NVIDIA-Nemotron-Nano-9B-v2` The NVIDIA Nemotron family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. Nemotron-Nano-9B-v2 is a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models.
NVIDIA Nemotron 3 Super (NVIDIA) nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 The NVIDIA Nemotron 3 Super is a 120B-parameter MoE model (12B active) delivering high-quality reasoning and generation for enterprise AI agents.
NVIDIA Nemotron 3 Nano (NVIDIA) nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 The NVIDIA Nemotron 3 Nano is a compact model designed for efficient edge and enterprise deployment with strong reasoning capabilities.
StarCoder2 (3B-15B) bigcode/starcoder2-7b StarCoder2 is a family of open large language models (LLMs) specialized for code generation and understanding. It is the successor to StarCoder, jointly developed by the BigCode project (a collaboration between Hugging Face, ServiceNow Research, and other contributors).
Jet-Nemotron jet-ai/Jet-Nemotron-2B Jet-Nemotron is a new family of hybrid-architecture language models that surpass state-of-the-art open-source full-attention language models, while achieving significant efficiency gains.
Trinity (Nano, Mini) arcee-ai/Trinity-Mini Arcee's foundational MoE Trinity family of models, open weights under Apache 2.0.
LFM2 (230M, 350M, 1.2B) LiquidAI/LFM2.5-1.2B-Instruct Liquid AI's hybrid language model combining gated short convolutions with a small number of grouped query attention (GQA) blocks.
LFM2-MoE (8B-A1B, 24B-A2B) LiquidAI/LFM2-8B-A1B Liquid AI's sparse Mixture-of-Experts variant of the LFM2 hybrid backbone, with SwiGLU experts, normalized sigmoid routing, and top-k expert selection.
Falcon-H1 (0.5B–34B) tiiuae/Falcon-H1-34B-Instruct TII's hybrid Mamba-Transformer architecture combining attention and state-space models for efficient long-context inference.
Hunyuan-Large (389B, MoE) tencent/Tencent-Hunyuan-Large Tencent's open-source MoE model with 389B total / 52B active parameters, featuring Cross-Layer Attention (CLA) for improved efficiency.
**IBM Granite 4.0 (Hybrid, Dense)** ibm-granite/granite-4.0-h-micro, ibm-granite/granite-4.0-micro IBM Granite 4.0 micro models: hybrid Mamba–MoE (h-micro) and dense (micro) variants. Enterprise-focused reasoning models
Granite SWA (IBM) ibm-granite/granite-swash-2b IBM Granite dense model with per-layer sliding window attention and learnable per-head attention sinks. Served by the Granite implementation; needs transformers ≥ 5.15.
Granite MoE SWA (IBM) ibm-granite/granite-swash-3b-a600m IBM Granite Mixture-of-Experts model with per-layer sliding window attention, learnable attention sinks, and optional shared experts. Served by the GraniteMoE implementation; needs transformers ≥ 5.15.
Sarvam 2 (30B-A2B, 105B-A10B) sarvamai/sarvam-2 Sarvam's Mixture-of-Experts models. The 105B variant uses MLA (Multi-head Latent Attention) and the 30B variant uses GQA, both with 128 routed experts.
Laguna XS.2 (poolside) poolside/Laguna-XS.2 Poolside's hybrid sliding-window-attention MoE model (256 routed experts, 1 shared expert, sigmoid router) with per-layer-type RoPE (YARN on full-attention layers, default on sliding-attention layers) and a softplus per-head attention gate.
Mellum 2 (JetBrains) JetBrains/Mellum2-12B-A2.5B-Base, JetBrains/Mellum2-12B-A2.5B-Thinking JetBrains' Qwen3-MoE-based code generation model with interleaved sliding-window/full attention, per-layer-type RoPE, and per-layer dense/sparse MLP routing.
# MindSpore Models Source: https://docs.sglang.io/docs/supported-models/mindspore_models ## Introduction MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang. ## Requirements MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN 8.5. The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). ## Supported Models Currently, the following models are supported: * **Qwen3**: Dense and MoE models * **DeepSeek V3/R1** * *More models coming soon...* ## Installation > **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../hardware-platforms/ascend-npus/getting-started/installation) and then install `sgl-mindspore`: ```bash Install theme={null} git clone https://github.com/mindspore-lab/sgl-mindspore.git cd sgl-mindspore pip install -e . ``` ## Run Model Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. ### Offline inference Use the following script for offline inference: ```python Offline Infer theme={null} import sglang as sgl # Initialize the engine with MindSpore backend llm = sgl.Engine( model_path="/path/to/your/model", # Local model path device="npu", # Use NPU device model_impl="mindspore", # MindSpore implementation attention_backend="ascend", # Attention backend tp_size=1, # Tensor parallelism size dp_size=1 # Data parallelism size ) # Generate text prompts = [ "Hello, my name is", "The capital of France is", "The future of AI is" ] sampling_params = {"temperature": 0, "top_p": 0.9} outputs = llm.generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"Prompt: {prompt}") print(f"Generated: {output['text']}") print("---") ``` ### Start server Launch a server with MindSpore backend: ```bash Command theme={null} # Basic server startup python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --tp-size 1 \ --dp-size 1 ``` For distributed server with multiple nodes: ```bash Command theme={null} # Multi-node distributed server python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --dist-init-addr 127.0.0.1:29500 \ --nnodes 2 \ --node-rank 0 \ --tp-size 4 \ --dp-size 2 ``` ## Troubleshooting #### Debug Mode Enable sglang debug logging by log-level argument. ```bash Debug Mode theme={null} python3 -m sglang.launch_server \ --model-path /path/to/your/model \ --host 0.0.0.0 \ --device npu \ --model-impl mindspore \ --attention-backend ascend \ --log-level DEBUG ``` Enable mindspore info and debug logging by setting environments. ```bash Command theme={null} export GLOG_v=1 # INFO export GLOG_v=0 # DEBUG ``` #### Explicitly select devices Use the following environment variable to explicitly select the devices to use. ```bash Command theme={null} export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device ``` #### Some communication environment issues In case of some environment with special communication environment, users need set some environment variables. ```bash Disable LCCL theme={null} export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore ``` #### Some dependencies of protobuf In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch. ```bash Command theme={null} export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch ``` ## Support For MindSpore-specific issues: * Refer to the [MindSpore documentation](https://www.mindspore.cn/) # Use Models From ModelScope Source: https://docs.sglang.io/docs/supported-models/modelscope To use a model from [ModelScope](https://www.modelscope.cn), set the environment variable `SGLANG_USE_MODELSCOPE`. ```bash Set Environment Variable theme={null} export SGLANG_USE_MODELSCOPE=true ``` We take [Qwen2-7B-Instruct](https://www.modelscope.cn/models/qwen/qwen2-7b-instruct) as an example. Launch the Server: ```bash Python theme={null} python -m sglang.launch_server --model-path qwen/Qwen2-7B-Instruct --port 30000 ``` Or start it by docker: ```bash Docker theme={null} docker run --gpus all \ -p 30000:30000 \ -v ~/.cache/modelscope:/root/.cache/modelscope \ --env "SGLANG_USE_MODELSCOPE=true" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 30000 ``` Note that modelscope uses a different cache directory than huggingface. You may need to set it manually to avoid running out of disk space. # Multimodal Language Models Source: https://docs.sglang.io/docs/supported-models/multimodal_language_models These models accept multi-modal inputs (e.g., images and text) and generate text output. They augment language models with multimodal encoders. ## Example launch Command ```bash Launch Server theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \ # example HF/local path --host 0.0.0.0 \ --port 30000 \ ``` > See the [OpenAI APIs section](../basic_usage/openai_api_vision) for how to send multimodal requests. ## Supported models Below the supported models are summarized in a table. If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen2_5_VLForConditionalGeneration`, use the expression: ```text GitHub Search theme={null} repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen2_5_VLForConditionalGeneration ``` in the GitHub search bar.
Model Family (Variants) Example HuggingFace Identifier Description Notes
Qwen-VL Qwen/Qwen3-VL-235B-A22B-Instruct Alibaba's vision-language extension of Qwen; for example, Qwen2.5-VL (7B and larger variants) can analyze and converse about image content.
DeepSeek-VL2 deepseek-ai/deepseek-vl2 Vision-language variant of DeepSeek (with a dedicated image processor), enabling advanced multimodal reasoning on image and text inputs.
DeepSeek-OCR / OCR-2 deepseek-ai/DeepSeek-OCR-2 OCR-focused DeepSeek models for document understanding and text extraction. Use --trust-remote-code.
Janus-Pro (1B, 7B) deepseek-ai/Janus-Pro-7B DeepSeek's open-source multimodal model capable of both image understanding and generation. Janus-Pro employs a decoupled architecture for separate visual encoding paths, enhancing performance in both tasks.
MiniCPM-V / MiniCPM-o openbmb/MiniCPM-V-2\_6 MiniCPM-V (2.6, \~8B) supports image inputs, and MiniCPM-o adds audio/video; these multimodal LLMs are optimized for end-side deployment on mobile/edge devices.
Llama 3.2 Vision (11B) meta-llama/Llama-3.2-11B-Vision-Instruct Vision-enabled variant of Llama 3 (11B) that accepts image inputs for visual question answering and other multimodal tasks.
LLaVA (v1.5 & v1.6) e.g. liuhaotian/llava-v1.5-13b Open vision-chat models that add an image encoder to LLaMA/Vicuna (e.g. LLaMA2 13B) for following multimodal instruction prompts.
LLaVA-NeXT (8B, 72B) lmms-lab/llava-next-72b Improved LLaVA models (with an 8B Llama3 version and a 72B version) offering enhanced visual instruction-following and accuracy on multimodal benchmarks.
LLaVA-OneVision lmms-lab/llava-onevision-qwen2-7b-ov Enhanced LLaVA variant integrating Qwen as the backbone; supports multiple images (and even video frames) as inputs via an OpenAI Vision API-compatible format.
Gemma 3 (Multimodal) google/gemma-3-4b-it Gemma 3's larger models (4B, 12B, 27B) accept images (each image encoded as 256 tokens) alongside text in a combined 128K-token context.
Kimi-VL (A3B) moonshotai/Kimi-VL-A3B-Instruct Kimi-VL is a multimodal model that can understand and generate text from images.
Mistral-Small-3.1-24B mistralai/Mistral-Small-3.1-24B-Instruct-2503 Mistral 3.1 is a multimodal model that can generate text from text or images input. It also supports tool calling and structured output.
Phi-4-multimodal-instruct microsoft/Phi-4-multimodal-instruct Phi-4-multimodal-instruct is the multimodal variant of the Phi-4-mini model, enhanced with LoRA for improved multimodal capabilities. It supports text, vision and audio modalities in SGLang.
MiMo-VL (7B) XiaomiMiMo/MiMo-VL-7B-RL Xiaomi's compact yet powerful vision-language model featuring a native resolution ViT encoder for fine-grained visual details, an MLP projector for cross-modal alignment, and the MiMo-7B language model optimized for complex reasoning tasks.
GLM-4.5V (106B) / GLM-4.1V(9B) zai-org/GLM-4.5V GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning Use --chat-template glm-4v
PaddleOCR-VL (0.9B, 1.5, 1.6) PaddlePaddle/PaddleOCR-VL-1.6 Baidu's 0.9B document-parsing VLM: a NaViT-style dynamic-resolution SigLIP encoder on an ERNIE-4.5-0.3B backbone, covering text, tables, formulas, charts and seals in 109 languages. See the cookbook page. Task is selected by the prompt (OCR:, Table Recognition:, ...). Leave --trust-remote-code off — transformers supports this architecture natively and its image processor is the faster one.
GLM-OCR zai-org/GLM-OCR GLM-OCR: A fast and accurate general OCR model
DotsVLM (General/OCR) rednote-hilab/dots.vlm1.inst RedNote's vision-language model built on a 1.2B vision encoder and DeepSeek V3 LLM, featuring NaViT vision encoder trained from scratch with dynamic resolution support and enhanced OCR capabilities through structured image data training.
DotsVLM-OCR rednote-hilab/dots.ocr Specialized OCR variant of DotsVLM optimized for optical character recognition tasks with enhanced text extraction and document understanding capabilities. Don't use --trust-remote-code
NVILA (8B, 15B, Lite-2B, Lite-8B, Lite-15B) Efficient-Large-Model/NVILA-8B chatml NVILA explores the full stack efficiency of multi-modal design, achieving cheaper training, faster deployment and better performance.
NVIDIA Nemotron Nano 2.0 VL nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 NVIDIA Nemotron Nano v2 VL enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q\&A and summarization capabilities. It builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, in order to achieve higher inference throughput in long document and video scenarios. Use --trust-remote-code. You may need to adjust --max-mamba-cache-size \[default is 512] to fit memory constraints.
Ernie4.5-VL baidu/ERNIE-4.5-VL-28B-A3B-PT Baidu's vision-language models(28B,424B). Support image and video comprehension, and also support thinking.
JetVLM JetVLM is an vision-language model designed for high-performance multimodal understanding and generation tasks built upon Jet-Nemotron. Coming soon
Step3-VL (10B) stepfun-ai/Step3-VL-10B StepFun's lightweight open-source 10B parameter VLM for multimodal intelligence, excelling in visual perception, complex reasoning, and human alignment.
Qwen3-ASR (0.6B, 1.7B) Qwen/Qwen3-ASR-1.7B Alibaba's automatic speech recognition models supporting 52 languages. Served via the /v1/audio/transcriptions endpoint.
Qwen3-Omni Qwen/Qwen3-Omni-30B-A3B-Instruct Alibaba's omni-modal MoE model. Currently supports the Thinker component (multimodal understanding for text, images, audio, and video), while the Talker component (audio generation) is not yet supported.
LFM2-VL LiquidAI/LFM2.5-VL-1.6B Liquid AI's vision-language model combining a SigLIP2 NaFlex vision encoder (variable resolution, native aspect ratio) with the LFM2 hybrid gated short conv + GQA language model. Supports multi-image inputs.
LocateAnything (3B) nvidia/LocateAnything-3B NVIDIA's visual grounding/detection model (MoonViT vision encoder + Qwen2 backbone) that emits \label\\...\ outputs with coordinates normalized to \[0, 1000]. Covers object detection, phrase grounding, scene-text detection, GUI grounding, and pointing. Use --trust-remote-code. Set skip\_special\_tokens=false so the \/\ grounding tokens survive in the output. Constrained \ decoding is opt-in and client-side: start the server with --enable-custom-logit-processor, then pass custom\_logit\_processor (a top-level request field) and custom\_params (inside sampling\_params) together — use LocateAnythingBoxGrammarLogitProcessor.build\_sampling\_params(config) to build both from the config token ids.
## Audio Transcription SGLang supports audio-only ASR models via the OpenAI-compatible `/v1/audio/transcriptions` endpoint. Upload an audio file and receive a transcription. ### Launch Command ```bash Command theme={null} sglang serve \ --model-path Qwen/Qwen3-ASR-1.7B \ --served-model-name qwen3-asr \ --trust-remote-code \ --host 0.0.0.0 --port 30000 ``` ### Example Request ```bash Command theme={null} curl http://localhost:30000/v1/audio/transcriptions \ -F file=@audio.wav \ -F model=qwen3-asr \ -F response_format=verbose_json ```
Model Family Example Identifier Notes
Whisper openai/whisper-large-v3 OpenAI's speech recognition model.
Qwen3-ASR (0.6B, 1.7B) Qwen/Qwen3-ASR-1.7B Use --trust-remote-code. Supports 52 languages.
## Video Input Support SGLang supports video input for Vision-Language Models (VLMs), enabling temporal reasoning tasks such as video question answering, captioning, and holistic scene understanding. Video clips are decoded, key frames are sampled, and the resulting tensors are batched together with the text prompt, allowing multimodal inference to integrate visual and linguistic context.
Model Family Example Identifier Video notes
Qwen-VL (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-Omni) Qwen/Qwen3-VL-235B-A22B-Instruct The processor gathers video\_data, runs Qwen's frame sampler, and merges the resulting features with text tokens before inference.
GLM-4v (4.5V, 4.1V, MOE) zai-org/GLM-4.5V Video clips are read with Decord, converted to tensors, and passed to the model alongside metadata for rotary-position handling.
NVILA (Full & Lite) Efficient-Large-Model/NVILA-8B The runtime samples eight frames per clip and attaches them to the multimodal request when video\_data is present.
LLaVA video variants (LLaVA-NeXT-Video, LLaVA-OneVision) lmms-lab/LLaVA-NeXT-Video-7B The processor routes video prompts to the LlavaVid video-enabled architecture, and the provided example shows how to query it with sgl.video(...) clips.
NVIDIA Nemotron Nano 2.0 VL nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 The processor samples at 2 FPS, at a max of 128 frames, as per model training. The model uses EVS, a pruning method that removes redundant tokens from video embeddings. By default video\_pruning\_rate=0.7. Change this by providing: --json-model-override-args '\{"video\_pruning\_rate": 0.0}' to disable EVS, for example.
JetVLM The runtime samples eight frames per clip and attaches them to the multimodal request when video\_data is present.
Use `sgl.video(path, num_frames)` when building prompts to attach clips from your SGLang programs. Example OpenAI-compatible request that sends a video clip: ```python Complete Example theme={null} import requests url = "http://localhost:30000/v1/chat/completions" data = { "model": "Qwen/Qwen3-VL-30B-A3B-Instruct", "messages": [ { "role": "user", "content": [ {"type": "text", "text": "What’s happening in this video?"}, { "type": "video_url", "video_url": { "url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4" }, }, ], } ], "max_tokens": 300, } response = requests.post(url, json=data) print(response.text) ``` ## Usage Notes ### Performance Optimization For multimodal models, you can use the `--keep-mm-feature-on-device` flag to optimize for latency at the cost of increased GPU memory usage: * **Default behavior**: Multimodal feature tensors are moved to CPU after processing to save GPU memory * **With `--keep-mm-feature-on-device`**: Feature tensors remain on GPU, reducing device-to-host copy overhead and improving latency, but consuming more GPU memory Use this flag when you have sufficient GPU memory and want to minimize latency for multimodal inference. ### Multimodal Inputs Limitation * **Use `--mm-process-config '{"image":{"max_pixels":1048576},"video":{"fps":3,"max_pixels":602112,"max_frames":60}}'`**: To set `image`, `video`, and `audio` input limits. This can reduce GPU memory usage, improve inference speed, and help to avoid OOM, but may impact model performance, thus set a proper value based on your specific use case. The config entries are passed as `images_kwargs`, `videos_kwargs`, and `audio_kwargs` to the HuggingFace processor, so each modality's settings are kept separate and do not collide. Refer to the HuggingFace documentation for your model's processor to understand the available parameters. ### Bidirectional Attention in Multimodal Model Serving **Note for serving the Gemma-3 multimodal model**: As mentioned in [Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM ](https://huggingface.co/blog/gemma3#multimodality), Gemma-3 employs bidirectional attention between image tokens during the prefill phase. Currently, SGLang only supports bidirectional attention when using the Triton Attention Backend. Note, however, that SGLang's current bidirectional attention implementation is incompatible with both CUDA Graph and Chunked Prefill. To enable bidirectional attention, you can use the `TritonAttnBackend` while disabling CUDA Graph and Chunked Prefill. Example launch command: ```bash Bidirectional Attention theme={null} python -m sglang.launch_server \ --model-path google/gemma-3-4b-it \ --host 0.0.0.0 --port 30000 \ --enable-multimodal \ --dtype bfloat16 --triton-attention-reduce-in-fp32 \ --attention-backend triton \ # Use Triton attention backend --disable-cuda-graph \ # Disable Cuda Graph --chunked-prefill-size -1 # Disable Chunked Prefill ``` If higher serving performance is required and a certain degree of accuracy loss is acceptable, you may choose to use other attention backends, and you can also enable features like CUDA Graph and Chunked Prefill for better performance, but note that the model will fall back to using causal attention instead of bidirectional attention. # Rerank models Source: https://docs.sglang.io/docs/supported-models/rerank_models SGLang offers comprehensive support for rerank models by incorporating optimized serving frameworks with a flexible programming interface. This setup enables efficient processing of cross-encoder reranking tasks, improving the accuracy and relevance of search result ordering. SGLang’s design ensures high throughput and low latency during reranker model deployment, making it ideal for semantic-based result refinement in large-scale retrieval systems. Rerank models in SGLang fall into two categories: * **Cross-encoder rerank models**: run with `--is-embedding` (embedding runner). * **Decoder-only rerank models**: run **without** `--is-embedding` and use next-token logprob scoring (yes/no). * Text-only (e.g. Qwen3-Reranker) * Multimodal (e.g. Qwen3-VL-Reranker): also supports image/video content Some models may require `--trust-remote-code`. ## Supported rerank models
Model Family (Rerank) Example HuggingFace Identifier Chat Template Description
BGE-Reranker (BgeRerankModel) BAAI/bge-reranker-v2-m3 N/A Currently only support attention-backend triton and torch\_native. High-performance cross-encoder reranker model from BAAI. Suitable for reranking search results based on semantic relevance.
Qwen3-Reranker (decoder-only yes/no) Qwen/Qwen3-Reranker-8B examples/chat\_template/qwen3\_reranker.jinja Decoder-only reranker using next-token logprob scoring for labels (yes/no). Launch without --is-embedding.
Qwen3-VL-Reranker (multimodal yes/no) Qwen/Qwen3-VL-Reranker-2B examples/chat\_template/qwen3\_vl\_reranker.jinja Multimodal decoder-only reranker supporting text, images, and videos. Uses yes/no logprob scoring. Launch without --is-embedding.
## Cross-Encoder Rerank (embedding runner) ### Launch Command ```shell theme={null} python3 -m sglang.launch_server \ --model-path BAAI/bge-reranker-v2-m3 \ --host 0.0.0.0 \ --disable-radix-cache \ --chunked-prefill-size -1 \ --attention-backend triton \ --is-embedding \ --port 30000 ``` ### Example Client Request ```python theme={null} import requests url = "http://127.0.0.1:30000/v1/rerank" payload = { "model": "BAAI/bge-reranker-v2-m3", "query": "what is panda?", "documents": [ "hi", "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China." ], "top_n": 1, "return_documents": True } response = requests.post(url, json=payload) response_json = response.json() for item in response_json: if item.get("document"): print(f"Score: {item['score']:.2f} - Document: '{item['document']}'") else: print(f"Score: {item['score']:.2f} - Index: {item['index']}") ``` **Request Parameters:** * `query` (required): The query text to rank documents against * `documents` (required): List of documents to be ranked * `model` (required): Model to use for reranking * `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned. * `return_documents` (optional): Whether to return documents in the response. Defaults to `True`. ## Qwen3-Reranker (decoder-only yes/no rerank) ### Launch Command ```shell theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-Reranker-0.6B \ --trust-remote-code \ --disable-radix-cache \ --host 0.0.0.0 \ --port 8001 \ --chat-template examples/chat_template/qwen3_reranker.jinja ``` Qwen3-Reranker uses decoder-only logprob scoring (yes/no). Do NOT launch it with `--is-embedding`. ### Example Client Request (supports optional instruct, top\_n, and return\_documents) ```shell theme={null} curl -X POST http://127.0.0.1:8001/v1/rerank \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-Reranker-0.6B", "query": "法国首都是哪里？", "documents": [ "法国的首都是巴黎。", "德国的首都是柏林。", "香蕉是黄色的水果。" ], "instruct": "Given a web search query, retrieve relevant passages that answer the query.", "top_n": 2, "return_documents": true }' ``` **Request Parameters:** * `query` (required): The query text to rank documents against * `documents` (required): List of documents to be ranked * `model` (required): Model to use for reranking * `instruct` (optional): Instruction text for the reranker * `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned. * `return_documents` (optional): Whether to return documents in the response. Defaults to `True`. ### Response Format `/v1/rerank` returns a list of objects (sorted by descending score): * `score`: float, higher means more relevant * `document`: the original document string (only included when `return_documents` is `true`) * `index`: the original index in the input `documents` * `meta_info`: optional debug/usage info (may be present for some models) The number of returned results is controlled by the `top_n` parameter. If `top_n` is not specified or is greater than the total number of documents, all documents are returned. Example (with `return_documents: true`): ```json theme={null} [ {"score": 0.99, "document": "法国的首都是巴黎。", "index": 0}, {"score": 0.01, "document": "德国的首都是柏林。", "index": 1}, {"score": 0.00, "document": "香蕉是黄色的水果。", "index": 2} ] ``` Example (with `return_documents: false`): ```json theme={null} [ {"score": 0.99, "index": 0}, {"score": 0.01, "index": 1}, {"score": 0.00, "index": 2} ] ``` Example (with `top_n: 2`): ```json theme={null} [ {"score": 0.99, "document": "法国的首都是巴黎。", "index": 0}, {"score": 0.01, "document": "德国的首都是柏林。", "index": 1} ] ``` ### Common Pitfalls * **`--chat-template` is required.** Without `--chat-template examples/chat_template/qwen3_reranker.jinja`, the server does not recognize the model as a decoder-only reranker and returns a 400 error: `"This model does not appear to be an embedding model by default. Please add `--is-embedding`..."`. The fix is to add the chat template flag, NOT `--is-embedding`. * If you launch Qwen3-Reranker with `--is-embedding`, `/v1/rerank` cannot compute yes/no logprob scores. Relaunch **without** `--is-embedding`. * If you see a validation error like "score should be a valid number" and the backend returned a list, upgrade to a version that coerces `embedding[0]` into `score` for rerank responses. ## Qwen3-VL-Reranker (multimodal decoder-only rerank) Qwen3-VL-Reranker extends the Qwen3-Reranker to support multimodal content, allowing reranking of documents containing text, images, and videos. ### Launch Command ```shell theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-Reranker-2B \ --trust-remote-code \ --disable-radix-cache \ --host 0.0.0.0 \ --port 30000 \ --chat-template examples/chat_template/qwen3_vl_reranker.jinja ``` Qwen3-VL-Reranker uses decoder-only logprob scoring (yes/no) like Qwen3-Reranker. Do NOT launch it with `--is-embedding`. ### Text-Only Reranking (backward compatible) ```python theme={null} import requests url = "http://127.0.0.1:30000/v1/rerank" payload = { "model": "Qwen3-VL-Reranker-2B", "query": "What is machine learning?", "documents": [ "Machine learning is a branch of artificial intelligence that enables computers to learn from data.", "The weather in Paris is usually mild with occasional rain.", "Deep learning is a subset of machine learning using neural networks with many layers.", ], "instruct": "Retrieve passages that answer the question.", "return_documents": True } response = requests.post(url, json=payload) results = response.json() for item in results: print(f"Score: {item['score']:.4f} - {item['document'][:60]}...") ``` ### Image Reranking (text query, image/mixed documents) ```python theme={null} import requests url = "http://127.0.0.1:30000/v1/rerank" payload = { "query": "A woman playing with her dog on a beach at sunset.", "documents": [ # Document 1: Text description "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset.", # Document 2: Image URL [ { "type": "image_url", "image_url": { "url": "https://example.com/beach_dog.jpeg" } } ], # Document 3: Text + Image (mixed) [ {"type": "text", "text": "A joyful scene at the beach:"}, { "type": "image_url", "image_url": { "url": "https://example.com/beach_dog.jpeg" } } ] ], "instruct": "Retrieve images or text relevant to the user's query.", "return_documents": False } response = requests.post(url, json=payload) results = response.json() for item in results: print(f"Index: {item['index']}, Score: {item['score']:.4f}") ``` ### Multimodal Query Reranking (query with image) ```python theme={null} import requests url = "http://127.0.0.1:30000/v1/rerank" payload = { # Query with text and image "query": [ {"type": "text", "text": "Find similar images to this:"}, { "type": "image_url", "image_url": { "url": "https://example.com/reference_image.jpeg" } } ], "documents": [ "A cat sleeping on a couch.", "A woman and her dog enjoying the sunset at the beach.", "A busy city street with cars and pedestrians.", [ { "type": "image_url", "image_url": { "url": "https://example.com/similar_image.jpeg" } } ] ], "instruct": "Find images or descriptions similar to the query image." } response = requests.post(url, json=payload) results = response.json() for item in results: print(f"Index: {item['index']}, Score: {item['score']:.4f}") ``` ### Request Parameters (Multimodal) * `query` (required): Can be a string (text-only) or a list of content parts: * `{"type": "text", "text": "..."}` for text * `{"type": "image_url", "image_url": {"url": "..."}}` for images * `{"type": "video_url", "video_url": {"url": "..."}}` for videos * `documents` (required): List where each document can be a string or list of content parts (same format as query) * `instruct` (optional): Instruction text for the reranker * `top_n` (optional): Maximum number of documents to return * `return_documents` (optional): Whether to return documents in the response (default: `false`) ### Common Pitfalls * Always use `--chat-template examples/chat_template/qwen3_vl_reranker.jinja` for Qwen3-VL-Reranker. * Do NOT launch with `--is-embedding`. * For best results, use `--disable-radix-cache` to avoid caching issues with multimodal content. * **Note**: Currently only `Qwen3-VL-Reranker-2B` is tested and supported. The 8B model may have different behavior and is not guaranteed to work with this template. # Reward models Source: https://docs.sglang.io/docs/supported-models/reward_models These models output a scalar reward score or classification result, often used in reinforcement learning or content moderation tasks. They are executed with `--is-embedding` and some may require `--trust-remote-code`. ## Example launch Command ```shell Command theme={null} python3 -m sglang.launch_server \ --model-path Qwen/Qwen2.5-Math-RM-72B \ # example HF/local path --is-embedding \ --host 0.0.0.0 \ --tp-size=4 \ # set for tensor parallelism --port 30000 \ ``` ## Supported models | Model Family (Reward) | Example HuggingFace Identifier | Description | | ------------------------------------------------------------------ | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | **Llama (3.1 Reward / `LlamaForSequenceClassification`)** | `Skywork/Skywork-Reward-Llama-3.1-8B-v0.2` | Reward model (preference classifier) based on Llama 3.1 (8B) for scoring and ranking responses for RLHF. | | **Gemma 2 (27B Reward / `Gemma2ForSequenceClassification`)** | `Skywork/Skywork-Reward-Gemma-2-27B-v0.2` | Derived from Gemma‑2 (27B), this model provides human preference scoring for RLHF and multilingual tasks. | | **InternLM 2 (Reward / `InternLM2ForRewardMode`)** | `internlm/internlm2-7b-reward` | InternLM 2 (7B)–based reward model used in alignment pipelines to guide outputs toward preferred behavior. | | **Qwen2.5 (Reward - Math / `Qwen2ForRewardModel`)** | `Qwen/Qwen2.5-Math-RM-72B` | A 72B math-specialized RLHF reward model from the Qwen2.5 series, tuned for evaluating and refining responses. | | **Qwen2.5 (Reward - Sequence / `Qwen2ForSequenceClassification`)** | `jason9693/Qwen2.5-1.5B-apeach` | A smaller Qwen2.5 variant used for sequence classification, offering an alternative RLHF scoring mechanism. | # How to Support New Models Source: https://docs.sglang.io/docs/supported-models/support_new_models This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations. This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations. ## How to Support a New Language Model To support a new model in SGLang, you only need to add a single file under the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn from existing model implementations and create a new file for your model. For most models, you should be able to find a similar model to start with (e.g., starting from Llama). Also refer how to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang) ## How to Support a New Multimodal Large Language Model To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the standard LLM support: 1. **Register your new model as multimodal**: Extend `is_multimodal_model` in [model\_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561) to return `True` for your model. 2. **Register a new chat-template**: Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function. 3. **Multimodal Data Processor**: Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your model’s dedicated processor. See [multimodal\_processor.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors) for more details. 4. **Handle Multimodal Tokens**: Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data with `RadixAttention`. 5. **Handle Image Feature Extraction**: Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model. 6. **Adapt to Vision Attention**: Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`. You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs. ## Testing and Debugging Please note all your testing and benchmarking results in PR description. ### Interactive Debugging For interactive debugging, compare the outputs of Hugging Face/Transformers and SGLang. The following two commands should give the same text output and very similar prefill logits: * Get the reference output: ```bash Command theme={null} python3 scripts/playground/reference_hf.py --model-path [new model] --model-type {text,vlm} ``` * Get the SGLang output: ```bash Command theme={null} python3 -m sglang.bench_one_batch --correct --model [new model] ``` ### Add the Model to the Test Suite To ensure the new model is well maintained, add it to the test suite by including it in the `ALL_OTHER_MODELS` list in the [test\_generation\_models.py](https://github.com/sgl-project/sglang/blob/main/test/registered/models/test_generation_models.py) file, test the new model on your local machine and report the results on demonstrative benchmarks (GSM8K, MMLU, MMMU, MMMU-Pro, etc.) in your PR. \\ For VLMs, also include a test in `test_vision_openai_server_{x}.py` (e.g. [test\_vision\_openai\_server\_a.py](https://github.com/sgl-project/sglang/blob/main/test/registered/vlm/test_vision_openai_server_a.py)). This is an example command to run to test a new model on your local machine: ```bash Run Test theme={null} ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerationModels.test_others ``` ### Benchmark * **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer\_guide/benchmark\_and\_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). * **(Optional) Other evals**: If you ran other evals, please note the results in PR description. ## Port a Model from vLLM to SGLang The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models from vLLM to SGLang. To port a model from vLLM to SGLang: * Compare these two files for guidance: * [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py) * [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py) * The major differences include: * **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`). * **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.** * **Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.** * **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers. * **Remove `Sample`.** * **Change the `forward()` functions** and add a `forward_batch()` method. * **Add `EntryClass`** at the end. * **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components. Note: make sure you add your new model to the supported models list in the supported models documentation. ## Registering an External Model Implementation In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server. This allows you to integrate your model without modifying the source code. For example: ```python Register Model theme={null} from sglang.srt.models.registry import ModelRegistry from sglang.srt.entrypoints.http_server import launch_server # For a single model, add it to the registry: ModelRegistry.models[model_name] = model_class # For multiple models, you can imitate the import_model_classes() function: from functools import lru_cache @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {} # Populate model_arch_name_to_cls with your new model classes. ... return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) # Launch the server with your server arguments: launch_server(server_args) ``` ## Example: Implementing and Serving a Llama Wrapper Model Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](../basic_usage/offline_engine_api). ### Implementing Our Model To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit. Let's start by defining our model in a file called `llama_wrapper.py`. The first step is to import the necessary libraries from SRT, which is SGLang's internal backend. ```python Example theme={null} # In the file `llama_wrapper.py` import torch from transformers import LlamaConfig from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM ``` Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`. Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219). Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us. ```python Class Definition theme={null} class LlamaWrapper(LlamaForCausalLM): def __init__( self, config: LlamaConfig, quant_config: Optional[QuantizationConfig] = None, prefix: str = "", ) -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) ``` Now, we want to define the `forward` method, which is what will be called at inference time. Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references. To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py). ```python Forward Method Signature theme={null} @torch.no_grad() def forward( self, input_ids: torch.Tensor, positions: torch.Tensor, forward_batch: ForwardBatch, pp_proxy_tensors: Optional[PPProxyTensors] = None, input_embeds: Optional[torch.Tensor] = None, get_embedding: bool = False, ) -> LogitsProcessorOutput: ``` We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method. After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`). ```python Call Model and LogitsProcessor theme={null} hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) ``` After receiving the logits for the next token, we can finally perform our biasing step. ```python Logit Biasing theme={null} orig_logits = res.next_token_logits res.next_token_logits = torch.where( orig_logits > 0, orig_logits.sqrt(), orig_logits ) return res ``` Now, our `LlamaWrapper` model is created and ready to be served! ### Serving Our Model Via SGLang's Offline Engine The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server. First, create a new file called `run.py`. Now, we must ensure that SGLang's `ModelRegistry` can find our model. To do this, we first download the model's configuration and weights from Huggingface. ```python Example theme={null} # In the file `run.py` import asyncio from functools import lru_cache from huggingface_hub import snapshot_download from llama_wrapper import LlamaWrapper # Make sure to import our new model! import sglang as sgl from sglang.srt.models.registry import ModelRegistry # Make sure to request access to this model on Huggingface, then export your # `HF_TOKEN` to download the model snapshot llama_dir = snapshot_download( repo_id="meta-llama/Llama-3.1-8B-Instruct", local_dir="./llama_ckpt", ) ``` Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`. That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model. ```python Example theme={null} { "architectures": [ # "LlamaForCausalLM" "LlamaWrapper" ], ... } ``` However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model. Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation". ```python Register LlamaWrapper theme={null} @lru_cache() def import_new_model_classes(): model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper} return model_arch_name_to_cls ModelRegistry.models.update(import_new_model_classes()) ``` Lastly, when we create our `Engine`, we just pass in the path to the local model directory. Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint. ```python Example theme={null} def main(): llm = sgl.Engine(model_path="./llama_ckpt") sampling_params = {"temperature": 0.2, "top_k": 5} prompts = [ "Write a short, neutral self-introduction for a fictional character. Hello, my name is", "Provide a concise factual statement about France’s capital city. The capital of France is", "Explain possible future trends in artificial intelligence. The future of AI is", ] asyncio.run(run_llm(llm, sampling_params, prompts)) llm.shutdown() async def run_llm( llm, sampling_params, prompts, ) -> None: outputs = await llm.async_generate(prompts, sampling_params) for prompt, output in zip(prompts, outputs): print(f"\nPrompt: {prompt}") print(f"Generated text: {output['text']}") if __name__ == "__main__": main() ``` Now, when we call `python run.py`, we will get the outputs of our newly created model! ## Serving External Models via the Standard CLI The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable. ### The `EntryClass` Variable When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`. For example, if you are implementing a Llama wrapper, add this line at the end of your model file: ```python Example theme={null} # This is what "Add EntryClass at the end" means EntryClass = LlamaWrapper ``` ### Example: Text-Only Model Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI. 1. Create your project ``` sglang_custom_project/ |----setup.py |----custom_llm/ |----__init__.py |----llama_wrapper.py ``` Write the `setup.py`: ```python Example theme={null} # sglang_custom_project/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins", version="0.1", packages=find_packages(), ) ``` 2. Write your model code Inside `llama_wrapper.py`, write your model and include `EntryClass`: ```python Example theme={null} # sglang_custom_project/custom_llm/llama_wrapper.py import torch from typing import Optional from sglang.srt.layers.logits_processor import LogitsProcessorOutput from sglang.srt.layers.quantization.base_config import QuantizationConfig from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors from sglang.srt.models.llama import LlamaForCausalLM class LlamaWrapper(LlamaForCausalLM): def __init__(self, config, quant_config: Optional[QuantizationConfig] = None, prefix: str = "") -> None: super().__init__(config=config, quant_config=quant_config, prefix=prefix) @torch.no_grad() def forward(self, input_ids, positions, forward_batch, pp_proxy_tensors=None, input_embeds=None, get_embedding=False): hidden_states = self.model( input_ids, positions, forward_batch, input_embeds, pp_proxy_tensors=pp_proxy_tensors, ) res: LogitsProcessorOutput = self.logits_processor( input_ids, hidden_states, self.lm_head, forward_batch, ) orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res # Don't forget to add EntryClass EntryClass = LlamaWrapper ``` 3. Install your package Run this inside your `sglang_custom_project` directory to install your code into the active Python environment: ```bash Command theme={null} pip install -e . ``` 4. Update your `config.json` Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name: ```json Config theme={null} { "architectures": ["LlamaWrapper"], ... } ``` 5. Launch the server Set the environment variable before running the CLI: ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm python -m sglang.launch_server \ --model-path /path/to/Llama-3.1-8B-Instruct \ --port 8000 ``` The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`. ### Example: Multimodal Model If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor. You can handle this by setting two additional environment variables: * `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models. * `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class. For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits. Create the project: ``` sglang_custom_project_vl/ |----setup.py |----custom_vlm/ |----__init__.py |----qwenvl_wrapper.py ``` Write `setup.py`: ```python Example theme={null} # sglang_custom_project_vl/setup.py from setuptools import setup, find_packages setup( name="sglang-custom-plugins-vl", version="0.1", packages=find_packages(), ) ``` Write the model in `qwenvl_wrapper.py`: ```python Example theme={null} # sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py import torch from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor class CustomQwen2VL(Qwen2VLForConditionalGeneration): def forward(self, input_ids, positions, forward_batch, input_embeds=None, get_embedding=False): res = super().forward( input_ids, positions, forward_batch, input_embeds=input_embeds, get_embedding=get_embedding ) if not get_embedding: orig = res.next_token_logits res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig) return res class CustomQwen2VLProcessor(QwenVLImageProcessor): models = [CustomQwen2VL] def __init__(self, hf_config, server_args, _processor, *args, **kwargs): super().__init__(hf_config, server_args, _processor, *args, **kwargs) EntryClass = CustomQwen2VL ``` **Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class. Install the package, update `config.json`, and launch: ```bash Command theme={null} pip install -e . ``` ```json Config theme={null} { "architectures": ["CustomQwen2VL"], ... } ``` ```bash Command theme={null} export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm python -m sglang.launch_server \ --model-path /path/to/Qwen2-VL-2B-Instruct \ --port 8000 \ --enable-multimodal ``` ## Documentation Add to table of supported models in [generative\_models.md](./generative_models) or [multimodal\_language\_models.md](./multimodal_language_models) *** By following these guidelines, you can add support for new language models and multimodal large language models in SGLang and ensure they are thoroughly tested and easily integrated into the system. # Transformers Fallback in SGLang Source: https://docs.sglang.io/docs/supported-models/transformers_fallback `sglang` can fall back to using models that are available in `transformers`. This works for most decoder-style language models and support for vision-language models is coming soon! ## Example launch Command By default, we will use sglang implementation if it is available. Otherwise, we will fall back to transformers one. However, you can switch the implementation by setting `--model-impl` to `transformers`. ```shell Launch Server theme={null} python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.2-1B-Instruct \ --host 0.0.0.0 \ --port 30000 \ --model-impl transformers ``` ## Supported features ### Quantization Transformers fall back has supported most of available quantization in SGLang (except GGUF). See [Quantization page](../advanced_features/quantization) for more information about supported quantization in SGLang. ### Remote code This fallback also means that any model on the hub that can be used in `transformers` with `trust_remote_code=True` that correctly implements attention can be used in production! A model just needs the following two things: ```python Example theme={null} from transformers import PreTrainedModel from torch import nn class MyAttention(nn.Module): def forward(self, hidden_states, **kwargs): # <- kwargs are required ... attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation] attn_output, attn_weights = attention_interface( self, query_states, key_states, value_states, **kwargs, ) ... class MyModel(PreTrainedModel): _supports_attention_backend = True ``` Here is what happens in the background: 1. The config is loaded 2. `MyModel` python class is loaded from the `auto_map`, and we check that the model `_supports_attention_backend`. 3. The `TransformersModel` backend is used. See `/srt/models/transformers`, which leverages `self.config._attn_implementation = "sglang"`, thus the need to use `ALL_ATTENTION_FUNCTIONS`. That's it! # Welcome to SGLang Source: https://docs.sglang.io/index High-performance serving framework for large language and multimodal models. Star Fork