> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Kimi-K3.1-Pro > Deploy Moonshot AI's Kimi K3.1 Pro with SGLang: 2.8T parameters (104B active) and a 1M context, on NVIDIA Vera Rubin, Blackwell and Hopper, AMD Instinct, PPU and Google TPU. ## Deployment > [accordion] Install SGLang > > Kimi K3.1 Pro is served from the `kimi-k31-pro` branch of [SGLang](https://github.com/sgl-project/sglang), based on `main`. To build and install from source, follow `docker/k31.Dockerfile` so that every dependency is in place. > > > [tab] Python (from source) > > > > ```bash Command theme={null} > > git clone -b kimi-k31-pro https://github.com/sgl-project/sglang.git && cd sglang > > pip install -e "python" > > ``` > > > [tab] Docker > > > > Use the Kimi K3.1 Pro image `lmsysorg/sglang:dev-kimi-k31-pro` (`lmsysorg/sglang:dev-kimi-k31-pro-mi35x` on AMD MI350X / MI355X, ROCm 10) and substitute the inner `sglang serve ...` with what the command generator below produces. See [Install → Method 3: Using Docker](https://docs.sglang.io/docs/get-started/install.md#method-3-using-docker). > > > [tab] PPU (M890 / 810E) > > > > Use the PPU image `egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun-tmp/inference-xpu-pytorch:kimi-k3.1-pro-sglang0.5.22-cuda13.0-pytorch2.13.0-20261103` with a prepared [SAIL environment](https://docs.sglang.io/docs/hardware-platforms/sail.md), and run the command panel's **Python** output inside the container on each worker node. > > > [tab] TPU (TPU7x) > > > > Install [SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu#get-started) on every host, then follow [TPU7x deployment](#tpu). Pick your hardware, deployment mode and serving strategy to generate the launch command: * **Low-Latency (TP8)**: the fastest replies with a few users at once. * **High-Concurrency (DCP8)** (B300, GB300, Unified): the most requests and context per GPU, with the KV cache split across the eight GPUs. * **Balanced** (decode role, other platforms): good speed with several users at once, for typical multi-user serving. * **High-Throughput**: the most tokens per second across many users, for batch jobs and many concurrent agent sessions. * **Prefill / Decode**: run the two phases on separate nodes (see [PD disaggregation](#pd-disaggregation)). The prefill role offers Low-Latency (TP8) and High-Throughput (PP4 × PCP2); the decode role offers Balanced (DCP8 · EP2) and High-Throughput (DCP8). On MI350X / MI355X, both roles offer High-Throughput (DCP8). Each command carries a badge: * **Verified**: that exact hardware and command were tested. * **Final Verification In Progress**: the recipe runs and is queued for measurement on the final weights. * **Not Verified** (Playground only): the Playground command does not match a Verified cell. > [!NOTE] > > The B300 `Decode` Balanced cell is **Verified** with DFLASH (1P1D round below, AIME26 ×8 = 1.000), and the B300 `Decode` High-Throughput cell without speculation. The other cells are still in verification. No cell has a cookbook-format speed sweep yet, so the benchmark card shows *pending*. > > The MI350X / MI355X cells are **Verified** with DFLASH; [AMD MI350X / MI355X](#amd-tips) says which to pick for your agent count. ### Mamba ratio calculator On `Unified`, `--mamba-full-memory-ratio` follows your average request length. The prefill and MI350X / MI355X High-Throughput recipes pin the ratio they were measured with; the decode and MI350X / MI355X Low-Latency recipes size the KDA pool directly, so the ratio does not apply. ## Advanced Features Playground Turn on additional knobs on top of whichever cell the Deploy panel is showing: * **Serving:** parsers, the PD transfer, HiCache storage and write policy, DFLASH proposed tokens, the prefill CUDA graph and the PD decode radix cache. * **WideEP Decode** (GB300 `Decode`): one decode server on four trays, attention DP2 with TP8 · DCP8 in each 8-GPU group and MegaMoE EP16 across all 16 GPUs. In a fixed-batch survey it beat 8-GPU decode workers from 8 concurrent requests up, by 15% at 64. That survey ran without the radix cache; turning the PD decode radix cache on still keeps one KDA state slot per request, since the decode tree caches KV only. Not Verified on the current code. * **Tuning:** the DSA attention backend and the MoE runner (B300, GB300, B200, GB200), the DCP communication backend, the KDA state slot count, KDA states per radix path, the MegaMoE token cap (when MoE Runner is MegaMoE) and the PP layer partition (PP4 × PCP2 prefill). ## Agentic Long-Context Deployment Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments on GB300 with Dynamo and SGLang, and on MI355X with SGLang, measured on the AgentX agentic-coding trace replay. Pick a deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order. > [!NOTE] > > These commands are a **status snapshot** (2026-11-03) of a GB300 tuning campaign that is still running. They are not an InferenceX submission. Each was measured with the DFlash acceptance length fixed to the draft's expected value (see the note under each configuration). The aggregated deployment at concurrency 1 missed its time-to-first-token target; the other seven points met theirs. * **Router.** Every configuration was measured behind Dynamo (`dynamo.frontend` with `dynamo.sglang` workers and the KV-aware router with session affinity). The SGLang router (`sglang_router` in front of `sglang.launch_server` workers) is derived from the same engine settings and was not measured. * **KV offload.** The disaggregated prefill workers were measured with HiCache in host DRAM; the aggregated workers without KV offload. The other tiers (none, HiCache, or a Mooncake store attached through the external linker) are derived and were not measured. * **Container image.** The commands show ``. The measured image was built with `docker/k31.Dockerfile` from SGLang commit `9ab3c28176` plus one unmerged change, and bundles FlashInfer 0.7.2, `sgl-deep-gemm` and a Dynamo 1.6.0 build with Kimi K3.1 Pro and session-affinity patches. `lmsysorg/sglang:dev-kimi-k31-pro` has not been checked against these commands. * **Draft.** The commands load the [Kimi-K3.1-Dflash2-Preview](https://huggingface.co/RadixArk/Kimi-K3.1-Dflash2-Preview) draft by its Hugging Face id; the measured runs read a local copy of it. On MI355X: * **Shape.** Up to 56 agents, one SGLang server on one node: TP8 for 1 and 4 agents, TP8 · DCP8 from 8; 1 to 8 agents also run the decode monokernel (`--enable-amd-monokernel` with `SGLANG_AMD_MONO_K31_WIDE=1`). `--max-running-requests` and `--cuda-graph-max-bs-decode` follow the agent count (64 at 56). At 92 agents, the Deploy panel's `High-Throughput` prefill and decode run as a 1P1D on two nodes behind the SGLang router. * **KV offload.** From 16 agents, HiCache in host DRAM (write-back), with the no-offload tier derived; up to 8 agents, none, since the monokernel does not accept KV offload. Mooncake has not run on MI355X. * **Container image.** The commands show ``. The measured image was a ROCm 10 SGLang image with the Kimi K3.1 ROCm port; `lmsysorg/sglang:dev-kimi-k31-pro-mi35x` has not been checked against these commands. The measured MI355X points; tok/s/user is 1000 / P90 inter-token latency and TTFT is p50 / p90:
Agents Shape tok/s/GPU tok/s/user TTFT (s) Prefix hit
16 1×8 8,868 60.2 0.65 / 1.9 96.4%
32 1×8 15,053 36.3 1.2 / 5.2 96.5%
48 1×8 17,818 28.6 2.0 / 10.4 96.2%
56 1×8 17,529 25.6 3.3 / 20.3 95.7%
92 1P1D, 2×8 17,883 30.6 3.7 / 10.3 94.4%
## 1. Model Introduction **Kimi K3.1 Pro** is Moonshot AI's open-weight, natively multimodal agentic model and the first model in the K3.1 series, post-trained from Kimi-K3 with large-scale joint reinforcement learning. It is a **2.8T-parameter Mixture-of-Experts model with 104B activated**: 93 layers (1 dense) combining **69 KDA + 24 Gated Sparse-MLA (DSA)** attention layers, **896 experts with 16 selected per token plus 2 shared**, SiTU-GLU activations, a 160K vocabulary, the MoonViT-V2 vision encoder (401M) for image input, and a **1,048,576-token** context. Weights are **MXFP4 with MXFP8 activations** (quantization-aware training). SGLang serves it as `KimiK31ForConditionalGeneration` and forces FP8 KV with page size 64. Kimi K3.1 Pro always thinks; set the depth with the top-level `reasoning_effort` field: `low`, `high` or `max` (default `max`). For multi-turn conversations and tool calls, pass the complete assistant message back in `messages` as-is, including `reasoning_content` and `tool_calls`, not just `content`. **Recommended generation:** `temperature=1.0`, `top_p=0.95` (fixed by the model; informational — do not hardcode in sample code). **License:** [Kimi K3.1 Pro License](https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LICENSE). **Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3.1-Pro) · [Tech Blog](https://www.kimi.ai/blog/kimi-k3-1-pro) · [API Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-1-pro-quickstart) · [DFLASH draft](https://huggingface.co/RadixArk/Kimi-K3.1-Dflash2-Preview). ## 2. Configuration Tips ### Recipe shapes * **GB300**: every recipe runs on two trays (2×4 GPUs over MNNVL); the command adds the multi-node flags. * **MI350X / MI355X**: one 8-GPU node per server; the 1P1D runs on two nodes (see [AMD MI350X / MI355X](#amd-tips)). * **PD Mode** * `Unified` serves prefill and decode together. * `Prefill` / `Decode` split them (see [PD disaggregation](#pd-disaggregation)). * **Prefill strategy** (B300, GB300) * `Low-Latency (TP8)` is flat TP8. * `High-Throughput (PP4 × PCP2)` is PP4 × PCP2 with MegaMoE EP2 (the cell carries MegaMoE; switch the MoE runner in the Advanced Features Playground): the most prefill tokens per second, and it serves the 1M window. * **Unified strategy** * `Low-Latency (TP8)` (B300, GB300, MI350X, MI355X) is flat TP8. * `High-Concurrency (DCP8)` (B300, GB300) is TP8 · DCP8 · EP2. * `High-Throughput (DCP8)` (MI350X, MI355X) is TP8 · DCP8 over the a2a backend. * `Balanced` (other platforms) is the placeholder shape for each platform; the parentheses give its layout: TP8 on Vera Rubin, TP16 · DCP16 on B200 and GB200, TP16 · EP16 on H200, TP32 · EP32 on H100, TP32 · EP8 on TPU7x. * **Decode strategy** * `Balanced (DCP8 · EP2)` is TP8 · DCP8 · EP2. * `High-Throughput (DCP8)` is the capacity-first decode: TP8 · DCP8, measured on B300 without speculation. * **Attention Backend** (B300, GB300, B200, GB200; in the Advanced Features Playground) * `FlashMLA` is the default DSA kernel and keeps the reference FP8 KV format. * `TRT-LLM` and `Prims-TS` add `--dsa-prefill-backend` / `--dsa-decode-backend`. They store the KV as plain FP8 and also run the query in FP8, so the numerics differ slightly from FlashMLA. * With PD disaggregation, launch the prefill and decode roles with the same backend. * **MoE Runner** (B300, GB300, B200, GB200; in the Advanced Features Playground) * `TRT-LLM` is the default: the FlashInfer trtllm-gen fused MXFP4 MoE, with no extra flag. * `MegaMoE` adds `--moe-a2a-backend megamoe`, which sets EP to the TP size. On the flat TP cells it also adds `--shared-experts-tp-size` and the `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` cap (16384 for prefill and unified, 4096 for decode). * MegaMoE has only been measured on the High-Throughput (PP4 × PCP2) prefill; TRT-LLM on the other cells. * MegaMoE reserves a symmetric workspace that grows with EP size × the token cap; lower `--mem-fraction-static` if startup runs out of memory. * **Spec Decode** * DFLASH with the [Kimi-K3.1-Dflash2-Preview](https://huggingface.co/RadixArk/Kimi-K3.1-Dflash2-Preview) draft. * Block 8; block 6 on the High-Throughput (DCP8) decode. * **HiCache** * Off by default; L2 and L3 keep DCP on Kimi K3.1 Pro. * On a PD decode server HiCache also turns on the decode radix cache. * On MI350X / MI355X, `L1+L2 (host)` is the default except on Low-Latency and the PD decode (see [AMD MI350X / MI355X](#amd-tips)). ### Tuning notes * **Memory.** The KDA state pool (FP32, \~52 MB per request per rank at TP8) sets the concurrency ceiling; the MLA + indexer KV (\~18.9 KB per token) fills the rest. TP and PCP replicate the MLA KV; DCP is the only axis that shards it. * **DFLASH.** Decode needs `--enable-linear-replayssm-spec` (the cells carry it) and an explicit `--max-running-requests`. The draft KV ring is on by default and adds \~15% target KV. * **MegaMoE.** Non-prefill roles must set `--max-running-requests`; do not add `--enable-symm-mem` (it deadlocks). * **PP prefill.** Keep `SGLANG_PP_LAYER_PARTITION=24,24,24,21` and `CUDA_MODULE_LOADING=EAGER` (both in the High-Throughput prefill cell). * **HiCache.** Use `--hicache-io-backend direct --hicache-mem-layout page_first_direct` with `--mamba-max-states-per-path 1` (the HiCache overlay adds them). ### AMD MI350X / MI355X * Requires ROCm 10; the recipes have not run on ROCm 7 images. * **Environment.** Every MI350X / MI355X cell sets `SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1`, as on [Kimi-K3](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3.md#amd-env): the AITER ROCm path, its K3 fused kernels, the FlyDSL MoE kernels and the SiTU v2 MoE with A8W4 activation quantization. * **Pick the recipe by agent count.** * 1 to 4: `Unified` `Low-Latency (TP8)`, on the monokernel. * 8 to 56: `Unified` `High-Throughput (DCP8)`; set `--max-running-requests` and `--cuda-graph-max-bs-decode` to the agent count (64 for 56). At 8 agents, add the monokernel and turn HiCache off. * Past 56: the 1P1D, `High-Throughput` prefill and decode on two nodes (see [On AMD MI355X](#pd-amd)). * **DFLASH** proposes 5 draft tokens (block 6) on the Triton draft attention backend, with ReplaySSM on every role but the PD prefill. * **Monokernel (1 to 8 agents).** `--enable-amd-monokernel` with `SGLANG_AMD_MONO_K31_WIDE=1` runs decode and verify steps through the persistent kernels of `sgl-amd-mono` (in the image); other steps use the standard forward. It takes TP8 or DCP8 with DFLASH but not HiCache, and SGLang stops at startup with the reasons if a setup is outside what it supports. The log shows `[k31 wide] first wide step` once it engages. * **HiCache.** `L1+L2 (host)` is the default: write-back, `--hicache-ratio 2.5` and `SGLANG_HICACHE_MAMBA_HOST_RATIO=12` (8 on the PD prefill). Low-Latency and the PD decode run without it. * On agentic replay the host KDA-state slots bound the prefix hit rate before the host KV does, so `SGLANG_HICACHE_MAMBA_HOST_RATIO` sizes them apart from `--hicache-ratio`. The recipes keep every KDA state per path; `--mamba-max-states-per-path 1` lowered the hit rate. * The host pools pin about 190 GB per GPU at DCP8, about 1.5 TB per node. `HSA_USERPTR_FOR_PAGED_MEM=0` backs them with `hipHostMalloc` (GTT memory, which never migrates); page migration in registered host memory stalls every GPU queue. * L3 has not run on MI350X / MI355X. * **Memory.** `SGLANG_ROCM_RUNTIME_RESERVE_GB` (10 at DCP8, 6 at TP8) keeps GPU memory free for the HIP queues' scratch, which is allocated outside torch. `--mem-fraction-static` 0.87 or 0.88 overflowed GTT at startup or ran slower on the single-node recipe. * **`--prefill-decode-interval 16`** runs 16 decode steps after each prefill batch before the next prefill. At 48 agents, 16 matched the throughput of 8 with 16% more interactivity; p90 TTFT rose from 5.2 s to 10.4 s. * **MI350X** runs the MI355X commands, single-node and 1P1D. ## 3. Advanced Usage ### 3.1 Reasoning The `kimi_k3` reasoning parser puts thinking in `message.reasoning_content` and the answer in `message.content`. > [accordion] Reasoning Example (Python) > > ```python Example theme={null} > from openai import OpenAI > > client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") > resp = client.chat.completions.create( > model="moonshotai/Kimi-K3.1-Pro", > messages=[{"role": "user", "content": "What is 15% of 240?"}], > reasoning_effort="high", # "low" | "high" | "max" (default max) > ) > msg = resp.choices[0].message > print("Reasoning:", getattr(msg, "reasoning_content", None)) > print("Answer:", msg.content) > ``` > [accordion] Example Output > > ```text Output theme={null} > Pending update... > ``` ### 3.2 Tool Calling The `kimi_k3` tool-call parser surfaces structured calls in `message.tool_calls`; print `reasoning_content` too, since the model thinks before calling. > [accordion] Tool Calling Example (Python) > > ```python Example theme={null} > from openai import OpenAI > > client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") > tools = [{ > "type": "function", > "function": { > "name": "get_weather", > "description": "Get the current weather for a city", > "parameters": { > "type": "object", > "properties": {"city": {"type": "string"}}, > "required": ["city"], > }, > }, > }] > resp = client.chat.completions.create( > model="moonshotai/Kimi-K3.1-Pro", > messages=[{"role": "user", "content": "What's the weather in Beijing?"}], > tools=tools, > ) > msg = resp.choices[0].message > print("Reasoning:", getattr(msg, "reasoning_content", None)) > print("Tool calls:", msg.tool_calls) > ``` > [accordion] Example Output > > ```text Output theme={null} > Pending update... > ``` ### 3.3 PD Disaggregation * **Transfer**: Mooncake (DCP rejects other backends on either role). Prefill listens on `30000`, decode on `30100`; the `8998` after `--prefill` must match `--disaggregation-bootstrap-port`. * **When using DFLASH, run it on both roles**: the prefill writes the draft KV ring that decode reads. * **Restart the prefill role whenever the decode role restarts**: prefill keeps the old decode's Mooncake mapping. > [accordion] Router > > Use the Kimi K3.1 Pro build of `sgl-router` (the stock `sglang-router` 0.3.2 lacks the K3 chat rendering). > > ```bash Command theme={null} > SGLANG_ROUTER_STREAM_IDLE_TIMEOUT_SECS=3600 \ > sgl-router \ > --pd-disaggregation \ > --prefill http://:30000 8998 \ > --decode http://:30100 \ > --host 0.0.0.0 --port 8000 \ > --policy power_of_two \ > --model-id moonshotai/Kimi-K3.1-Pro \ > --tokenizer-path \ > --request-timeout-secs 540 > ``` **Measured 1P1D** (B300 2×8, SWA1024 draft, `High-Throughput` prefill without the prefill CUDA graph + `Decode` Balanced):
Workload Result
Prefill, 90% prefix hit, TTFT p90 ≤ 15 s 362,093 tok/s at concurrency 32
Decode, 8K context, batch 64 2,583 tok/s (TPOT p90 26.07 ms, AL 2.77)
Decode, 8K context, concurrency 1 TPOT 4.89 ms (AL 2.96); 9.6 ms without speculation
AIME26 ×8 1.000
#### On AMD MI355X * **Container**: add `--device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 --network=host` to the usual ROCm flags; Mooncake runs over RDMA and finds the RoCE NICs itself. * **Prefill host tier**: HiCache's host pools are pinned and count against the node's GTT limit, so a tier larger than that fails at startup. If yours does, lower `--hicache-ratio` or `SGLANG_HICACHE_MAMBA_HOST_RATIO`. * **Router**: the panel's MI355X router (`sglang_router` from the ROCm image) is the one measured. **Measured 1P1D** (MI355X 2×8, `High-Throughput` prefill + `High-Throughput` decode, AgentX at concurrency 92 for 3600 s, acceptance length simulated at 3.399):
Workload Result
AgentX, concurrency 92, 3600 s 17,883 tok/s per GPU at 30.6 tok/s per user (P90 ITL 32.7 ms)
Time to first token P50 3.66 s, P90 10.26 s (94.4% prefix hit)
### 3.4 TPU7x deployment Kimi K3.1 Pro runs on a 16-chip TPU7x slice (four hosts) as one TP32 · EP8 engine through [SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu). Run its recipe on each host: ```bash Command theme={null} python3 -m benchmark.tpu.launch_server --config recipe/configs/kimi_k3_1/server.py ``` Single request, all 93 layers: 133 tok/s at 8K input (TPOT 7.52 ms) and 112 tok/s at 128K. With the experimental decode megakernel and DFlash, single-user decode reaches up to about 590 tok/s. DFlash ([`RadixArk/Kimi-K3.1-Dflash2-Preview`](https://huggingface.co/RadixArk/Kimi-K3.1-Dflash2-Preview)) serves one request at a time. Structured output, tool-call grammar and image input are not yet supported on TPU7x. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.