GLM-5.3

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang

docker pull lmsysorg/sglang:latest

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.
• High-Throughput — most tokens per second across many users. Best for batch jobs.

​

Playground

​

1. Model Introduction

​

2. Configuration Tips

• DeepSeek Sparse Attention (DSA). GLM-5.3 uses the glm_moe_dsa architecture; SGLang auto-selects the DSA attention backends (flashmla_sparse prefill, fa3 decode, sgl-kernel indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — fp8_e4m3 on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and bf16 on Hopper (H200) — so no --kv-cache-dtype flag is required. On Hopper, pairing --kv-cache-dtype fp8_e4m3 with --dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3’s 64 query heads match the kernel’s native tile) — see the DeepSeek-V3.2 page for kernel details; the optional SGLANG_ENABLE_DSA_Q8KV8_* performance env vars are documented in python/sglang/srt/environ.py.
• MTP / speculative decoding. The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 for low-latency; 1-1-2 for balanced). The config’s index_share_for_mtp_iteration reuses the DSA indexer’s topk across draft steps (effective only at --speculative-eagle-topk 1). Watch the server’s reported accept length and adjust --speculative-num-steps / --speculative-num-draft-tokens: lower the draft length when rejected draft tokens create excess verification work.
• Memory. The FP8 weights are large (MoE total, not active params). Start around --mem-fraction-static 0.8 on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).
• DP-Attention + DeepEP for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.
• BF16 weights need more GPUs. The full-precision build (zai-org/GLM-5.3-BF16, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on 8×B300 (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a multi-node layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8.
• PD Disaggregation (prefill/decode). GLM-5.3 is a DSA model and runs under prefill/decode disaggregation — toggle the PD Disagg card in the Playground above (pick a Prefill/Decode role + transfer backend, then front the roles with sglang_router.launch_router --pd-disaggregation). The Mooncake backend auto-detects the InfiniBand HCA, so no device flag is needed by default; only add --disaggregation-ib-device mlx5_0 (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (--privileged --ulimit memlock=-1, or --device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK) — without IB exposure Mooncake silently falls back to TCP.
• Chunked-prefill size is regime-dependent. For long-input balanced workloads, start with --chunked-prefill-size 32768 and tune it together with --max-running-requests for your input length and KV capacity. Keep the default chunked-prefill size for the high-throughput recipe unless profiling shows that prefill is the bottleneck.
• AMD GPUs (MI300X / MI325X / MI355X). FP8 (zai-org/GLM-5.3) runs single-node at tp=8 on all three. BF16 (zai-org/GLM-5.3-BF16, ~1.51 TB) only fits single-node on MI325X (2 TB HBM) and MI355X (2.3 TB); MI300X (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there. Use the DSA tilelang backend (--dsa-prefill-backend tilelang --dsa-decode-backend tilelang) and add --chunked-prefill-size 131072 plus --watchdog-timeout 1200 (20 min for weight loading). GLM-5.3 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the DeepSeek-V3.2 cookbook.
• MTP / EAGLE speculative decoding is disabled for AMD in the Deploy panel because the gfx950 spec-decode draft kernel is not yet validated on this hardware (and at --speculative-num-steps > 3 hits a separate build issue). Until MTP is validated on gfx950, omit the --speculative-* flags and serve without MTP.

​

3. Advanced Usage

​

3.1 Reasoning

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[{"role": "user", "content": "What is 15% of 240?"}],
    extra_body={"chat_template_kwargs": {"clear_thinking": True, "reasoning_effort": "high"}},
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Answer:", msg.content)

Reasoning: 1.  **Identify the core question:** The user wants to find 15% of 240.
2.  **Convert the percentage to a decimal:** 15% = 0.15
3.  **Multiply by the total:** 0.15 * 240 = 36
    (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)

Answer: 15% of 240 is **36**.

Here is how you can calculate it:
0.15 × 240 = 36

​

3.2 Tool Calling

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="zai-org/GLM-5.3",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
)
msg = resp.choices[0].message
print("Reasoning:", getattr(msg, "reasoning_content", None))
print("Tool calls:", msg.tool_calls)

Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.

Tool calls: [
  {
    "id": "call_13fcd52146934b7781d06d4a",
    "type": "function",
    "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
  }
]

​

3.3 HiCache (Hierarchical KV Caching)

​

3.4 Claude Code Integration

export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
export ANTHROPIC_AUTH_TOKEN="dummy"
export API_TIMEOUT_MS="3000000"
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.3[1m]"
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.3[1m]"
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.3[1m]"
claude

• CLAUDE_CODE_ATTRIBUTION_HEADER=0 — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.3’s chat template renders tools before system, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
• glm-5.3[1m] as the model name — the [1m] suffix is the client-side hint that enables Claude Code’s 1M-context beta, matching GLM-5.3’s 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the model field, so any name is accepted server-side.

​

3.5 Context Parallelism

--attn-cp-size 8 \
--enable-prefill-cp \
--cp-strategy interleave \

--enable-dsa-cache-layer-split \
--enable-prefill-cp \
--attn-cp-size 8 \
--cp-strategy interleave \