GLM-5.3-Flash - SGLang Documentation

Deployment

Install SGLang

Use an SGLang build that includes GLM-5.3-Flash support.

Command

docker pull lmsysorg/sglang:glm-5.3-flash

The deployment panel can render a complete docker run command for the selected hardware and options. See Install SGLang with Docker for host setup.

Choose your hardware, then choose the operating point that matches your workload:

Low Latency starts with adaptive MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
High Throughput starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.

Every listed hardware platform exposes both strategies. A Verified badge means that exact hardware and command were tested. Not Verified means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported. The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to Not Verified without hiding the option.

Playground

Use the Playground for lower-level tuning such as attention parallelism, MoE communication, and reasoning or tool parsers. It inherits every selection from the deployment panel and shows only the command-line diff.

1. Model introduction

GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a hybrid attention architecture — 320B total parameters with 18B active. Its 45 text layers combine MLA attention, DSA sparse attention, and KDA linear attention, and a 24-layer vision encoder handles image and video input. The checkpoint uses 288 routed experts with 8 active experts per token and includes a native MTP draft layer for speculative decoding. See the GLM-5.3-Flash blog and the GLM-5 technical report for training details.

Attribute	Description
Architecture	MoE with hybrid attention (MLA, DSA, KDA), mHC, and MTP — 320B total / 18B active parameters
Precision	FP8 weights with BF16 KV cache by default
Context	1M tokens
Inputs	Text, image, and video
Generation defaults	`temperature=1.0`, `top_p=0.95`, thinking enabled

The deployment recipes use the checkpoint’s generation configuration. Override sampling only when your application has its own evaluated settings.

2. Configuration tips

Choose a strategy

Start with Low Latency for chat and agent workloads. Adaptive MTP changes the draft depth as acceptance changes, reducing unnecessary draft work when the server is busy. Measure High Throughput for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through --speculative-algorithm NEXTN, so generated commands use that flag value. Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes.

Size both memory pools

GLM-5.3-Flash maintains a paged KV pool for attention and a separate KDA state pool. The KDA state pool can limit concurrency before the KV pool is full. If startup reduces max_running_requests because of KDA state capacity, increase --mamba-full-memory-ratio or set --max-mamba-cache-size for the expected concurrency, then tune --max-running-requests to the workload. Keep the prefix cache enabled for every strategy. Keep the checkpoint’s KDA lower-bound setting unchanged. In particular, do not override linear_lower_bound through --json-model-override-args.

Keep the KV and DSA backends paired

The default recipe pairs a BF16 KV cache with TileLang DSA. On Blackwell GPUs, the deployment panel also offers FP8 KV with TRT-LLM DSA as a memory-saving option. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination. FP8 KV with TRT-LLM DSA is disabled on H100 and H200.

Extend the cache hierarchy

Keep HiCache off when GPU memory is sufficient. Select L1 + L2 to spill reusable cache entries into host memory. Select + L3 only after configuring Mooncake on every serving node; the generated command exposes the required configuration path. These options remain selectable but are marked Not Verified until the resulting command is validated on the chosen hardware.

Multimodal memory

All strategies enable multimodal serving. The processor samples video at 2 FPS and caps video input at 240,000 visual tokens. Install torchcodec in the serving environment before sending video requests. For very long videos on 4x GB300, use encoder disaggregation to isolate the vision encoder’s memory spikes from language decoding. The default multimodal feature transport is automatic, and on a single CUDA node auto resolves to CPU transport. CUDA IPC is opt-in: pass --mm-feature-transport cuda_ipc when lower transfer latency matters more than the GPU memory the IPC pool reserves. CUDA VMM transport applies only to multi-node GB200/GB300 systems on the MNNVL fabric, where auto selects it.

3. Advanced usage

3.1 Reasoning

Thinking is enabled by the checkpoint’s generation configuration, and generated commands enable --reasoning-parser glm45 by default. The OpenAI-compatible API then places thinking in message.reasoning_content and the final answer in message.content. You can disable Reasoning Parser in the Playground when an integration needs the raw response format. To disable thinking for a request, pass chat_template_kwargs: {"thinking": false} in the request body.

3.2 Tool calling

Generated commands enable --tool-call-parser glm47 by default, so structured calls are returned in message.tool_calls. You can disable Tool Call Parser in the Playground when tool calling is not needed. On follow-up turns, read both reasoning_content and content because a thinking model can use either field around tool execution.

3.3 Multimodal serving

The base recipes accept image and video content through the OpenAI-compatible chat API. Keep the processor defaults unless you have measured a different sampling or resize policy. Inputs above the video token budget are clamped to the processor’s limit rather than rejected.

3.4 Encoder disaggregation

Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4/EP4 process on port 30000. Start the encoder first.

Encoder server (GB300)

Command

sglang serve \
  --model-path zai-org/GLM-5.3-Flash \
  --tp-size 4 \
  --encoder-only \
  --host 0.0.0.0 \
  --port 30001

Language server (GB300)

Command

sglang serve \
  --model-path zai-org/GLM-5.3-Flash \
  --tp-size 4 \
  --ep-size 4 \
  --attention-backend dsa \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --linear-attn-backend triton \
  --kv-cache-dtype bfloat16 \
  --quantization fp8 \
  --moe-runner-backend deep_gemm \
  --max-running-requests 64 \
  --chunked-prefill-size 8192 \
  --max-prefill-tokens 8192 \
  --disable-shared-experts-fusion \
  --disable-prefill-cuda-graph \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-adaptive \
  --language-only \
  --encoder-urls http://localhost:30001 \
  --mem-fraction-static 0.78 \
  --host 0.0.0.0 \
  --port 30000

This topology served image requests and videos up to 238,080 visual tokens. In a concurrent long-video workload, the largest observed decode gap fell from 5.53 seconds in unified serving to 1.79 seconds with encoder disaggregation. Keep --mem-fraction-static 0.78 on the language process so the encoder retains room for vision workspaces. Install torchcodec for video and see the encoder disaggregation guide for the generic architecture and operational model.

3.5 PD disaggregation (preview)

PD splits prefill and decode into separate server groups behind a router. For this hybrid model, the transfer moves both the paged DSA KV and the KDA recurrent state.

PD serving is mechanically validated with dummy weights only — startup, bootstrap, state transfer, and request flow all work on 4x GB300. It has not been load- or accuracy-tested. Treat it as a preview until the real-weight gate completes.

PD serving on 4x GB300 (single node)

Prefill (GPU 0-1)

sglang serve \
  --model-path zai-org/GLM-5.3-Flash \
  --tp-size 2 \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype bfloat16 \
  --moe-runner-backend triton \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 8998 \
  --disaggregation-transfer-backend nixl \
  --host 0.0.0.0 \
  --port 31000

Decode (GPU 2-3)

sglang serve \
  --model-path zai-org/GLM-5.3-Flash \
  --tp-size 2 \
  --base-gpu-id 2 \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --kv-cache-dtype bfloat16 \
  --moe-runner-backend triton \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend nixl \
  --host 0.0.0.0 \
  --port 32000

Router

python -m sglang_router.launch_router \
  --pd-disaggregation \
  --mini-lb \
  --prefill http://127.0.0.1:31000 8998 \
  --decode http://127.0.0.1:32000 \
  --host 0.0.0.0 \
  --port 30000

The positional 8998 after --prefill must equal the prefill server’s --disaggregation-bootstrap-port.

Operational notes:

Give each role a distinct --nccl-port when both share one node.
Single-node NIXL needs UCX_NET_DEVICES=lo and UCX_TLS=tcp,cuda_copy,cuda_ipc,self,sm in both server environments.
The validated arm used the triton MoE runner; deep_gemm under PD is untested.

Known limitations:

Speculative decoding does not start under PD at the current cut (draft-graph capture width assert on the prefill role; decode-side memory pressure at TP2). Run PD without speculative flags.
Prefill and decode with different TP sizes transfer state through slice paths, but numeric correctness is unverified. Keep both roles at the same TP size.

​Deployment

​Playground

​1. Model introduction

​2. Configuration tips

​Choose a strategy

​Size both memory pools

​Keep the KV and DSA backends paired

​Extend the cache hierarchy

​Multimodal memory

​3. Advanced usage

​3.1 Reasoning

​3.2 Tool calling

​3.3 Multimodal serving

​3.4 Encoder disaggregation

​3.5 PD disaggregation (preview)

Deployment

Playground

1. Model introduction

2. Configuration tips

Choose a strategy

Size both memory pools

Keep the KV and DSA backends paired

Extend the cache hierarchy

Multimodal memory

3. Advanced usage

3.1 Reasoning

3.2 Tool calling

3.3 Multimodal serving

3.4 Encoder disaggregation

3.5 PD disaggregation (preview)