Kimi-K3.1-Pro

​

Deployment

git clone -b kimi-k31-pro https://github.com/sgl-project/sglang.git && cd sglang
pip install -e "python"

• Low-Latency (TP8): the fastest replies with a few users at once.
• High-Concurrency (DCP8) (B300, GB300, Unified): the most requests and context per GPU, with the KV cache split across the eight GPUs.
• Balanced (decode role, other platforms): good speed with several users at once, for typical multi-user serving.
• High-Throughput: the most tokens per second across many users, for batch jobs and many concurrent agent sessions.
• Prefill / Decode: run the two phases on separate nodes (see PD disaggregation). The prefill role offers Low-Latency (TP8) and High-Throughput (PP4 × PCP2); the decode role offers Balanced (DCP8 · EP2) and High-Throughput (DCP8). On MI350X / MI355X, both roles offer High-Throughput (DCP8).

• Verified: that exact hardware and command were tested.
• Final Verification In Progress: the recipe runs and is queued for measurement on the final weights.
• Not Verified (Playground only): the Playground command does not match a Verified cell.

​

Mamba ratio calculator

​

Advanced Features Playground

• Serving: parsers, the PD transfer, HiCache storage and write policy, DFLASH proposed tokens, the prefill CUDA graph and the PD decode radix cache.
• WideEP Decode (GB300 Decode): one decode server on four trays, attention DP2 with TP8 · DCP8 in each 8-GPU group and MegaMoE EP16 across all 16 GPUs. In a fixed-batch survey it beat 8-GPU decode workers from 8 concurrent requests up, by 15% at 64. That survey ran without the radix cache; turning the PD decode radix cache on still keeps one KDA state slot per request, since the decode tree caches KV only. Not Verified on the current code.
• Tuning: the DSA attention backend and the MoE runner (B300, GB300, B200, GB200), the DCP communication backend, the KDA state slot count, KDA states per radix path, the MegaMoE token cap (when MoE Runner is MegaMoE) and the PP layer partition (PP4 × PCP2 prefill).

​

Agentic Long-Context Deployment

• Router. Every configuration was measured behind Dynamo (dynamo.frontend with dynamo.sglang workers and the KV-aware router with session affinity). The SGLang router (sglang_router in front of sglang.launch_server workers) is derived from the same engine settings and was not measured.
• KV offload. The disaggregated prefill workers were measured with HiCache in host DRAM; the aggregated workers without KV offload. The other tiers (none, HiCache, or a Mooncake store attached through the external linker) are derived and were not measured.
• Container image. The commands show <K31_IMAGE>. The measured image was built with docker/k31.Dockerfile from SGLang commit 9ab3c28176 plus one unmerged change, and bundles FlashInfer 0.7.2, sgl-deep-gemm and a Dynamo 1.6.0 build with Kimi K3.1 Pro and session-affinity patches. lmsysorg/sglang:dev-kimi-k31-pro has not been checked against these commands.
• Draft. The commands load the Kimi-K3.1-Dflash2-Preview draft by its Hugging Face id; the measured runs read a local copy of it.

• Shape. Up to 56 agents, one SGLang server on one node: TP8 for 1 and 4 agents, TP8 · DCP8 from 8; 1 to 8 agents also run the decode monokernel (--enable-amd-monokernel with SGLANG_AMD_MONO_K31_WIDE=1). --max-running-requests and --cuda-graph-max-bs-decode follow the agent count (64 at 56). At 92 agents, the Deploy panel’s High-Throughput prefill and decode run as a 1P1D on two nodes behind the SGLang router.
• KV offload. From 16 agents, HiCache in host DRAM (write-back), with the no-offload tier derived; up to 8 agents, none, since the monokernel does not accept KV offload. Mooncake has not run on MI355X.
• Container image. The commands show <K31_MI35X_IMAGE>. The measured image was a ROCm 10 SGLang image with the Kimi K3.1 ROCm port; lmsysorg/sglang:dev-kimi-k31-pro-mi35x has not been checked against these commands.

​

1. Model Introduction

​

2. Configuration Tips

​

Recipe shapes

• GB300: every recipe runs on two trays (2×4 GPUs over MNNVL); the command adds the multi-node flags.
• MI350X / MI355X: one 8-GPU node per server; the 1P1D runs on two nodes (see AMD MI350X / MI355X).
• PD Mode
  • Unified serves prefill and decode together.
  • Prefill / Decode split them (see PD disaggregation).
• Prefill strategy (B300, GB300)
  • Low-Latency (TP8) is flat TP8.
  • High-Throughput (PP4 × PCP2) is PP4 × PCP2 with MegaMoE EP2 (the cell carries MegaMoE; switch the MoE runner in the Advanced Features Playground): the most prefill tokens per second, and it serves the 1M window.
• Unified strategy
  • Low-Latency (TP8) (B300, GB300, MI350X, MI355X) is flat TP8.
  • High-Concurrency (DCP8) (B300, GB300) is TP8 · DCP8 · EP2.
  • High-Throughput (DCP8) (MI350X, MI355X) is TP8 · DCP8 over the a2a backend.
  • Balanced (other platforms) is the placeholder shape for each platform; the parentheses give its layout: TP8 on Vera Rubin, TP16 · DCP16 on B200 and GB200, TP16 · EP16 on H200, TP32 · EP32 on H100, TP32 · EP8 on TPU7x.
• Decode strategy
  • Balanced (DCP8 · EP2) is TP8 · DCP8 · EP2.
  • High-Throughput (DCP8) is the capacity-first decode: TP8 · DCP8, measured on B300 without speculation.
• Attention Backend (B300, GB300, B200, GB200; in the Advanced Features Playground)
  • FlashMLA is the default DSA kernel and keeps the reference FP8 KV format.
  • TRT-LLM and Prims-TS add --dsa-prefill-backend / --dsa-decode-backend. They store the KV as plain FP8 and also run the query in FP8, so the numerics differ slightly from FlashMLA.
  • With PD disaggregation, launch the prefill and decode roles with the same backend.
• MoE Runner (B300, GB300, B200, GB200; in the Advanced Features Playground)
  • TRT-LLM is the default: the FlashInfer trtllm-gen fused MXFP4 MoE, with no extra flag.
  • MegaMoE adds --moe-a2a-backend megamoe, which sets EP to the TP size. On the flat TP cells it also adds --shared-experts-tp-size and the SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK cap (16384 for prefill and unified, 4096 for decode).
  • MegaMoE has only been measured on the High-Throughput (PP4 × PCP2) prefill; TRT-LLM on the other cells.
  • MegaMoE reserves a symmetric workspace that grows with EP size × the token cap; lower --mem-fraction-static if startup runs out of memory.
• Spec Decode
  • DFLASH with the Kimi-K3.1-Dflash2-Preview draft.
  • Block 8; block 6 on the High-Throughput (DCP8) decode.
• HiCache
  • Off by default; L2 and L3 keep DCP on Kimi K3.1 Pro.
  • On a PD decode server HiCache also turns on the decode radix cache.
  • On MI350X / MI355X, L1+L2 (host) is the default except on Low-Latency and the PD decode (see AMD MI350X / MI355X).

​

Tuning notes

• Memory. The KDA state pool (FP32, ~52 MB per request per rank at TP8) sets the concurrency ceiling; the MLA + indexer KV (~18.9 KB per token) fills the rest. TP and PCP replicate the MLA KV; DCP is the only axis that shards it.
• DFLASH. Decode needs --enable-linear-replayssm-spec (the cells carry it) and an explicit --max-running-requests. The draft KV ring is on by default and adds ~15% target KV.
• MegaMoE. Non-prefill roles must set --max-running-requests; do not add --enable-symm-mem (it deadlocks).
• PP prefill. Keep SGLANG_PP_LAYER_PARTITION=24,24,24,21 and CUDA_MODULE_LOADING=EAGER (both in the High-Throughput prefill cell).
• HiCache. Use --hicache-io-backend direct --hicache-mem-layout page_first_direct with --mamba-max-states-per-path 1 (the HiCache overlay adds them).

​

AMD MI350X / MI355X

• Requires ROCm 10; the recipes have not run on ROCm 7 images.
• Environment. Every MI350X / MI355X cell sets SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1, as on Kimi-K3: the AITER ROCm path, its K3 fused kernels, the FlyDSL MoE kernels and the SiTU v2 MoE with A8W4 activation quantization.
• Pick the recipe by agent count.
  • 1 to 4: Unified Low-Latency (TP8), on the monokernel.
  • 8 to 56: Unified High-Throughput (DCP8); set --max-running-requests and --cuda-graph-max-bs-decode to the agent count (64 for 56). At 8 agents, add the monokernel and turn HiCache off.
  • Past 56: the 1P1D, High-Throughput prefill and decode on two nodes (see On AMD MI355X).
• DFLASH proposes 5 draft tokens (block 6) on the Triton draft attention backend, with ReplaySSM on every role but the PD prefill.
• Monokernel (1 to 8 agents). --enable-amd-monokernel with SGLANG_AMD_MONO_K31_WIDE=1 runs decode and verify steps through the persistent kernels of sgl-amd-mono (in the image); other steps use the standard forward. It takes TP8 or DCP8 with DFLASH but not HiCache, and SGLang stops at startup with the reasons if a setup is outside what it supports. The log shows [k31 wide] first wide step once it engages.
• HiCache. L1+L2 (host) is the default: write-back, --hicache-ratio 2.5 and SGLANG_HICACHE_MAMBA_HOST_RATIO=12 (8 on the PD prefill). Low-Latency and the PD decode run without it.
  • On agentic replay the host KDA-state slots bound the prefix hit rate before the host KV does, so SGLANG_HICACHE_MAMBA_HOST_RATIO sizes them apart from --hicache-ratio. The recipes keep every KDA state per path; --mamba-max-states-per-path 1 lowered the hit rate.
  • The host pools pin about 190 GB per GPU at DCP8, about 1.5 TB per node. HSA_USERPTR_FOR_PAGED_MEM=0 backs them with hipHostMalloc (GTT memory, which never migrates); page migration in registered host memory stalls every GPU queue.
  • L3 has not run on MI350X / MI355X.
• Memory. SGLANG_ROCM_RUNTIME_RESERVE_GB (10 at DCP8, 6 at TP8) keeps GPU memory free for the HIP queues’ scratch, which is allocated outside torch. --mem-fraction-static 0.87 or 0.88 overflowed GTT at startup or ran slower on the single-node recipe.
• --prefill-decode-interval 16 runs 16 decode steps after each prefill batch before the next prefill. At 48 agents, 16 matched the throughput of 8 with 16% more interactivity; p90 TTFT rose from 5.2 s to 10.4 s.
• MI350X runs the MI355X commands, single-node and 1P1D.

​

3. Advanced Usage

​

3.1 Reasoning

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="moonshotai/Kimi-K3.1-Pro",
    messages=[{"role": "user", "content": "What is 15% of 240?"}],
    reasoning_effort="high",  # "low" | "high" | "max" (default max)
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)

Pending update...

​

3.2 Tool Calling

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="moonshotai/Kimi-K3.1-Pro",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Tool calls:", msg.tool_calls)

Pending update...

​

3.3 PD Disaggregation

• Transfer: Mooncake (DCP rejects other backends on either role). Prefill listens on 30000, decode on 30100; the 8998 after --prefill must match --disaggregation-bootstrap-port.
• When using DFLASH, run it on both roles: the prefill writes the draft KV ring that decode reads.
• Restart the prefill role whenever the decode role restarts: prefill keeps the old decode’s Mooncake mapping.

SGLANG_ROUTER_STREAM_IDLE_TIMEOUT_SECS=3600 \
sgl-router \
  --pd-disaggregation \
  --prefill http://<prefill-host>:30000 8998 \
  --decode http://<decode-host>:30100 \
  --host 0.0.0.0 --port 8000 \
  --policy power_of_two \
  --model-id moonshotai/Kimi-K3.1-Pro \
  --tokenizer-path <Kimi-K3.1-Pro tokenizer.json> \
  --request-timeout-secs 540

​

On AMD MI355X

• Container: add --device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 --network=host to the usual ROCm flags; Mooncake runs over RDMA and finds the RoCE NICs itself.
• Prefill host tier: HiCache’s host pools are pinned and count against the node’s GTT limit, so a tier larger than that fails at startup. If yours does, lower --hicache-ratio or SGLANG_HICACHE_MAMBA_HOST_RATIO.
• Router: the panel’s MI355X router (sglang_router from the ROCm image) is the one measured.

​

3.4 TPU7x deployment

python3 -m benchmark.tpu.launch_server --config recipe/configs/kimi_k3_1/server.py