DeepSeek-V4

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang

docker pull lmsysorg/sglang:latest

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    sglang serve <use args below>

docker pull lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016

docker run \
    --device=/dev/kfd --device=/dev/dri \
    --group-add video \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --shm-size 32g --ipc=host \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016 \
    sglang serve <use args below>

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.
• High-Throughput — most tokens per second across many users. Best for batch jobs.

​

Playground

• Built-in SGLang features — parallelism overrides (TP / CP / DP-Attention — DP-Attention’s value is the DP degree, with off to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
• DeepSeek-V4 specific features — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).

​

Agentic Long-Context Deployment

• Router. Each configuration was submitted behind one front door: Dynamo (dynamo.frontend with dynamo.sglang workers) or SGLang (sglang.launch_server, behind sglang_router when there are several workers or DP-attention ranks).
• KV offload. Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes --enable-unified-cache-external-linker.
• Container image. Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

​

1. Model Introduction

​

2. Configuration Tips

• low-latency: steps=3, draft-tokens=4 → largest win at bs=1.
• balanced: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
• high-throughput: MTP disabled — at saturation the verify step costs more than it saves.
• MTP runs on the v2 speculative path.

• Docker image — all three cells use lmsysorg/sglang:dev-v4f-2dgx-v2, a preview build made only for DGX Spark (branch b12x-vision @ 452239a74f): it bakes in the SM12x b12x MoE (W4A8) and compressed-MLA attention kernels (#34878, #35899, #34018), the Flash Vision model support (#37253), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel’s Docker mode — the bare Python command needs the b12x kernel package this image ships.
• Run the same command on both Sparks with --node-rank 0 / --node-rank 1 and --dist-init-addr pointing at node 0 over the ConnectX-7 link. The docker run flags the panel emits (--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%.
• Env knobs in the cells are part of the recipe: SGLANG_SM120_FLASHMLA_BACKEND=b12x selects the b12x attention path, SGLANG_B12X_MAX_TOKENS must equal --chunked-prefill-size, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True avoids unified-memory fragmentation OOMs on GB10.
• NVFP4 (nvidia/DeepSeek-V4-Flash-0731-NVFP4) — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint’s native formats. On SM12x that means three extra flags: --moe-runner-backend flashinfer_cutlass (b12x’s MoE is MXFP4-only and trtllm-gen kernels are sm100-only), --speculative-moe-runner-backend b12x (the DSpark draft’s MTP experts are MXFP4 and run on b12x), and --disable-shared-experts-fusion (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise.
• Flash Vision — images are served natively on the b12x recipe with the same flags as Flash Official (send image_url content on /v1/chat/completions, see Vision); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (~3.2 vs ~3.9) — with text accuracy intact.

• Preview build required — support lands via sgl-project/sglang#37253 and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image lmsysorg/sglang:dev-dsv4-flash-vision; for a Python environment, install SGLang from that PR’s branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image lmsysorg/sglang:dev-v4f-2dgx-v2 (see the DGX Spark notes).
• Verified matrix — MMMU-Pro via sgl-eval at temperature 1.0, top-p 0.95, --reasoning-effort max.
• Engine auto-configuration — the engine picks the flashinfer_mxfp4 MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don’t pass --enforce-shared-experts-fusion.
• Chunked prefill & radix cache stay enabled — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.
• Speculative decoding — the checkpoint bundles a DSpark head, and the low-latency recipes enable it with --speculative-algorithm DSPARK (verified with image batches on B200 via the MMMU-Pro round; the other hardware rows are pending verification). The balanced and high-throughput recipes run target-only: they use DP Attention, which DSpark is incompatible with on current releases. As on the 0731/0813 checkpoints, do not pass the EAGLE flags.

--enforce-shared-experts-fusion

SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  <other args>

--moe-a2a-backend deepep \
--deepep-mode auto \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# prefill
--moe-a2a-backend deepep \
--deepep-mode normal \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# decode
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# Please use the latest main branch for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-deepseek-v4-fp4-indexer

# Please use the latest ROCm image for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --tp 8 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  <other args from the Deploy panel>

sglang serve \
  --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
  --tp 8

sglang serve \
  --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
  --tp 8

• Original FP4 checkpoints — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the W4A8 path — MXFP4 weights with FP8 activations via FlashInfer’s Humming kernels — by adding --flashinfer-mxfp4-moe-precision fp8 to --moe-runner-backend flashinfer_mxfp4; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
• Converted FP8 checkpoints (H100 and H200 only) — pre-repackaged FP8 weights at sgl-project/DeepSeek-V4-Flash-FP8 and sgl-project/DeepSeek-V4-Pro-FP8 unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).

• Model checkpoints — for correct accuracy, the FP4 model uses the stock deepseek-ai/DeepSeek-V4-{Flash,Pro}, and the FP8 model uses the repackaged sgl-project/DeepSeek-V4-{Flash,Pro}-FP8.
• Supported models — MI300X supports DeepSeek-V4-Flash in FP8; MI355X supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
• TP / DP setting (MI355X) — both TP=4 and TP=8 are supported. At low concurrency we recommend TP-only; at high concurrency use TP + DP (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set --attn-dp-size 8 --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10.
• MTP — speculative decoding on the original Flash / Pro checkpoints, which bundle an MTP head; add --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4. Pro Official (0813) has no MTP head, so these flags do not apply there and the Playground does not offer them.
• DSpark (MI355X Pro Official 0813) — the 0813 checkpoint bundles the DSpark draft head, and DSpark is the only speculative algorithm it supports. Enable --speculative-algorithm DSPARK (low-latency) and see the MI355X agentic recipe for the long-context path. DSpark also runs under PD disaggregation — see the Pro Official pairs in §3.7.
• Kernels — uses the Unified KV attention and the flydsl MoE.
• FP4 indexer (MI355X) — FP4 C4 indexer is supported via --enable-deepseek-v4-fp4-indexer on top of the standard ROCm recipe.

export SGLANG_MORI_DISPATCH_DTYPE=mxfp8
export SGLANG_MORI_RECV_BOUND=1

sglang serve \
  --model-path {MODEL_PATH} \
  --tp 8 --attn-dp-size 8 \
  --ep-size 8 --moe-a2a-backend mori --deepep-mode normal \
  <other args from the Deploy panel>

• FP4: enable both env vars.
• FP8: use SGLANG_MORI_RECV_BOUND=1 only; omit SGLANG_MORI_DISPATCH_DTYPE=mxfp8.

• W4A8 — default MegaMoE kernel (FP4 weights, FP8 activations).
• W4A4 — adds --enable-w4a4-mxfp4-megamoe to run the custom W4A4 kernel (FP4 activations). The flag configures the required DeepGEMM settings. Higher throughput with negligible accuracy drop (~89.5 GPQA on Pro).

• The W4A8 / W4A4 variants above are Blackwell-only (B200 / B300 / GB200 / GB300). On Hopper (SM90, H100 / H200) use the all-FP8 MegaMoE path described below instead.
• MegaMoE is only wired into the high-throughput recipe on Blackwell (per sgl-project/sglang#26451). The chip is hidden on low-latency and balanced — switch to high-throughput to expose it.
• When running MegaMoE, don’t set --moe-runner-backend manually.
• Adjust SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).

SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
SGLANG_DSV4_FP4_EXPERTS=0 \
sglang serve \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 8 \
  --moe-a2a-backend megamoe \
  --chunked-prefill-size 4096

​

3. Advanced Usage

​

3.1 Reasoning

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if delta.content:
        if has_thinking and not has_answer:
            print("\n=============== Content =================", flush=True)
            has_answer = True
        print(delta.content, end="", flush=True)

print()

We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:

**Step 1: Understand the meaning of percent**
"Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).

**Step2: Convert the percentage to a decimal or fraction**
\( 15\% = \frac{15}{100} = 0.15 \)

**Step3: Multiply by the given number**
Multiply the decimal form by 240:
\( 0.15 \times 240 \)

**Step4: Perform the multiplication**
\( 0.15 \times 240 = 36 \)

**Answer:** 15% of 240 is **36**.

​

3.2 Tool Calling

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
    extra_body={"chat_template_kwargs": {"thinking": True}},
    stream=True,
)

thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta

    if getattr(delta, "reasoning_content", None):
        if not thinking_started:
            print("=============== Thinking =================", flush=True)
            thinking_started = True
        has_thinking = True
        print(delta.reasoning_content, end="", flush=True)

    if getattr(delta, "tool_calls", None):
        if has_thinking and thinking_started:
            print("\n=============== Content =================\n", flush=True)
            thinking_started = False
        for tool_call in delta.tool_calls:
            index = tool_call.index
            if index not in tool_calls_accumulator:
                tool_calls_accumulator[index] = {"name": None, "arguments": ""}
            if tool_call.function:
                if tool_call.function.name:
                    tool_calls_accumulator[index]["name"] = tool_call.function.name
                if tool_call.function.arguments:
                    tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments

    if delta.content:
        print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()

The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>

<｜DSML｜tool_calls>
<｜DSML｜invoke name="get_weather">
<｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
</｜DSML｜invoke>
</｜DSML｜tool_calls>

​

3.3 HiCache (Hierarchical KV Caching)

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file / mooncake / hf3fs / nixl); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

• L2 (GPU + CPU) — leave Storage on auto (default). Cold KV pages spill to CPU pinned memory only. Use direct IO backend + page_first_direct or layer-first mem-layout.
• L3 (GPU + CPU + Storage) — pick a Storage backend (file); the Playground emits the canonical page_first_direct mem-layout + direct IO backend + wait_complete prefetch policy, matching the HiCache best-practices recipe.

​

3.4 DSpark (Speculative Decoding)

sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \
  --host 0.0.0.0 \
  --port 30000

​

3.5 Vision (Image Inputs)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",
             "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
            {"type": "text", "text": "Describe this image in a few sentences."},
        ],
    }],
)

message = response.choices[0].message
if getattr(message, "reasoning_content", None):
    print("=============== Thinking =================")
    print(message.reasoning_content)
print("=============== Content =================")
print(message.content)

Pending update...

​

3.6 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)

• --cuda-graph-backend-prefill breakable at concurrency 1–4.
• --enable-prefill-cp --cp-strategy interleave --enable-cp-decode-attn-tp on the TP-only commands at concurrency 16 and above. Prefill context parallelism splits each prompt across all eight ranks (CP size = TP size).
• SGLANG_DSV4_GROUPED_ASM=1 on the TP-only commands at concurrency 16 and above — the kernel needs at least 16 requests per draft token.
• At concurrency 384 on the MegaMoE command: --prefill-decode-interval 20, SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=1024, and AITER_MEGA_FIXED_SLOT_MAX_MTPR=2047.

SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --prefill-decode-interval 20 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.86 \
  --chunked-prefill-size 16384 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --watchdog-timeout 3600

  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct

SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_DP_USE_REDUCE_SCATTER=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --attn-dp-size 8 \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 20 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --load-balance-method total_requests \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 131072 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct \
  --watchdog-timeout 3600

SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=16384 \
SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=512 \
AITER_MEGA_FIXED_SLOT_MAX_MTPR=1023 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=17179869184 \
SGLANG_AITER_MEGA_RANK_SYNC=1 \
SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \
SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \
SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE=7 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
GPU_MAX_HW_QUEUES=5 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --attn-dp-size 8 \
  --ep 8 \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 40 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --load-balance-method total_requests \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-a2a-backend megamoe \
  --moe-dense-tp-size 1 \
  --disable-shared-experts-fusion \
  --enable-eplb \
  --eplb-rebalance-num-iterations 200 \
  --expert-distribution-recorder-mode stat \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 131072 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct \
  --watchdog-timeout 3600

​

3.7 PD Disaggregation on MI355X (MORI-IO)

docker run \
    --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \
    --group-add video --cap-add IPC_LOCK \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    --ulimit nofile=1048576:1048576 \
    --network host --ipc=host --shm-size 32g \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261019 \
    python3 -m sglang.launch_server <role args below>

​

Role-based configuration

• CUDA graphs. Prefill runs --disable-cuda-graph, because the FP4 indexer’s prefill path is not captured. Decode captures --cuda-graph-bs-decode.
• Speculative decoding is the same on both. Keep --speculative-dspark-block-size equal on the two roles. The original Flash and Pro checkpoints have no DSpark head and use their MTP head instead: --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.

GPU_MAX_HW_QUEUES=5 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=8G \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --host 0.0.0.0 --port 6100 \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 7200 \
  --disaggregation-transfer-backend mori \
  --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \
  --tp-size 8 --dp-size 8 --ep-size 8 \
  --enable-dp-attention \
  --enable-dp-attention-local-control-broadcast \
  --enable-dp-lm-head \
  --moe-a2a-backend megamoe \
  --moe-dense-tp-size 1 \
  --enforce-shared-experts-fusion \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --kv-cache-dtype fp8_e4m3 \
  --page-size 256 \
  --swa-full-tokens-ratio 0.15 \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 65536 \
  --max-running-requests 512 \
  --context-length 1048576 \
  --disable-cuda-graph \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 3 \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-unified-cache-external-linker \
  --unified-cache-external-linker-backend mori \
  --optimistic-prefill-attempts 2 \
  --enable-cache-report \
  --load-balance-method round_robin \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --enable-metrics \
  --watchdog-timeout 3600

GPU_MAX_HW_QUEUES=5 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_DSV4_DENSE_MXSCALE=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
MORI_MAX_DISPATCH_TOKENS_DECODE=256 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --host 0.0.0.0 --port 6100 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mori \
  --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \
  --tp-size 8 --dp-size 8 \
  --enable-dp-attention \
  --enable-dp-attention-local-control-broadcast \
  --enable-dp-lm-head \
  --enforce-shared-experts-fusion \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --kv-cache-dtype fp8_e4m3 \
  --page-size 256 \
  --swa-full-tokens-ratio 0.15 \
  --mem-fraction-static 0.92 \
  --max-running-requests 512 \
  --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 \
  --context-length 1048576 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 3 \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --load-balance-method round_robin \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --enable-metrics \
  --watchdog-timeout 3600

• SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 is required. --moe-a2a-backend megamoe alone runs the DeepGEMM MegaMoE path, which has no ROCm kernel. Do not set this variable on decode: the scheduler reads it too, so it also changes how decode schedules batches.
• SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR must be at least --chunked-prefill-size / attn_dp_size, here 65536 / 8. Below that SGLang reports no error and runs fused MoE for the extra tokens. 8192 is the default, so the command omits this variable. Set it explicitly if you increase the per-rank chunk size.
• MORI_SHMEM_HEAP_SIZE sizes the MORI symmetric heap, where MegaMoEv2 allocates its dispatch and combine buffers. It needs about 0.51 GiB per 1024 tokens of MTPR — about 4.2 GiB at 8192 — and the 4 GiB default fails during CUDA graph capture. SGLang allocates the whole heap at the first MegaMoE forward pass, after the KV cache pool, so it consumes the GPU memory --mem-fraction-static left free whether MegaMoE fills it or not. 8 GiB fits next to 0.92; §3.6 runs at 0.85 and can afford more.

​

Router configuration

SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests \
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://<prefill-host>:30000 \
  --decode http://<decode-host>:30100 \
  --host 0.0.0.0 --port 8000 \
  --policy consistent_hashing --dp-aware \
  --decode-policy power_of_two \
  --worker-startup-check-interval 1 \
  --cache-threshold 0.3 \
  --balance-abs-threshold 2 --balance-rel-threshold 1.1 \
  --disable-circuit-breaker --health-failure-threshold 100 \
  --health-check-timeout-secs 600 --health-check-interval-secs 30

​

Additional Pro Official concurrency points

export SGLANG_AITER_MEGA_RANK_SYNC=1
export SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1
export SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1

--enable-eplb
--eplb-rebalance-num-iterations 200
--expert-distribution-recorder-mode stat

​

3.8 Unified Cache External Linker (UMBP)

--enable-unified-cache-external-linker \
--unified-cache-external-linker-backend mori

mkdir -p /tmp/umbp
UMBP_DRAM_CAPACITY=1500000000000 \
UMBP_DRAM_USE_HUGEPAGES=1 \
UMBP_SSD_ENABLED=0 \
MORI_UMBP_LOG_LEVEL=info \
<mori>/umbp_standalone_server unix:///tmp/umbp/standalone.grpc.sock

UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \
sglang serve ...