Deployment
Install SGLang
Install SGLang
Start an interactive container with GPU access, shared memory, the Hugging Face cache, port 30000, and your checkpoint token. Replace At the resulting container shell, run the Python panel’s
<your-hf-token> before running this command:Command
sglang serve command. Alternatively, run the Docker panel’s complete docker run command directly on the host. Keep the server running while you send requests to port 30000.- Low-Latency — DFly2 speculative decoding with the draft layers bundled in the checkpoint. Pick it for interactive traffic such as single-user chat and agents that decode greedily; see DFly2 requirements for the request restrictions.
- High-Throughput — no speculative decoding. Pick it for multi-user and batch traffic, for sampled decoding and grammar-constrained tool calls, and as the base for DP-Attention, context parallelism and the FP8 KV cache.
Checkpoints
| Checkpoint | Weights | Recipes on this page |
|---|---|---|
tencent/Hy4-FP8 | ~786 GB, block FP8 | 8× H200/B200 at TP8, 4× B300/GB300 at TP4 |
tencent/Hy4 | ~1.55 TB, BF16 | None: the weights exceed TP8 on H200/B200 and TP4 on B300/GB300 |
tencent/Hy4-FP8 stores the routed experts, the attention output and gate projections, and most of the DFly2 layers’ linear projections in block FP8: e4m3 weights in 128×128 blocks with dynamically quantized group-128 activations. The remaining attention projections, the DSA indexer, the first layer’s attention and dense MLP, the shared experts, lm_head and the vision tower stay in BF16. SGLang reads the quantization from the checkpoint, so the recipes pass no --quantization flag.
Both checkpoints declare a 1,048,576-token context. No recipe pins --context-length, so the server uses the declared length; the KV pool, not the declared length, bounds how many tokens fit at once.
Model capabilities
Hy4 pairs a 770B-parameter language backbone (MLA + DSA sparse attention + 256-expert MoE + iHC) with a 40-layer vision transformer and a Conv2d spatial-merge adapter, for image and video understanding with dynamic resolution (32-pixel alignment, 512²–2048² budget) and 3-channel mRoPE position encoding. The 770B count excludes the ViT, the adapter and the DFly2 layers.- Images: single and multiple images per request, in any position relative to text. Both the OpenAI chat API (
image_urlparts) and raw prompts with<image>placeholders work. - Video: sampled-frame video input (fps=2) with per-frame timestamp tokens. Long or high-resolution clips can exhaust the KV pool, so leave headroom for long videos.
- Tool calling + reasoning:
--reasoning-parser auto --tool-call-parser autoresolve to thehunyuanparsers.tool_choice: "required", a named function, and"auto"withstricttools are enforced by a native structural grammar: the call envelope, tool names and argument keys are constrained, while argument values are free text and are not validated against the schema. A named function orparallel_tool_calls: falseallows exactly one call. DFly2 rejects these grammar-constrained requests; serve them from the High-Throughput recipe. - DFly2 speculative decoding: the checkpoint bundles five DFly2 draft layers. One draft forward runs the bonus token, mixed with the target’s final hidden state, plus six mask tokens, and one target pass verifies the seven-token block. Each draft layer keeps its KV in the pool of a target layer and reuses a target indexer’s sparse-attention top-k, so the draft builds no indexer and no KV pool of its own. The seven-token block comes from the checkpoint’s
dfly_block_size, so the recipes pass no block-size flag. With the server defaults, the decode CUDA graph resolves tofulland captures the draft forward together with the target verify, the target prefill replays through abreakableCUDA graph, and overlap scheduling stays on.
Playground
The Playground adds knobs on top of the selected Deploy cell: DP-Attention and prefill context parallelism (CP2) on the High-Throughput recipe, the hierarchical KV cache, and the speculative preset. Everything else is inherited from the base recipe.DFly2 requirements
The server validates DFly2 before loading weights and refuses to start with any of:- a decode CUDA graph backend other than
fullordisabled, a prefill CUDA graph backend other thanbreakableordisabled, two-batch overlap or mixed chunked prefill; - DP-Attention, pipeline parallelism, prefill context parallelism (
--attn-cp-sizeabove 1), decode context parallelism (--dcp-sizeabove 1) or PD disaggregation; - a model dtype other than BF16, or a KV cache dtype other than BF16;
- LoRA;
- a separate draft checkpoint (
--speculative-draft-model-path), a draft attention backend, or a draft quantization or KV cache dtype that differs from the target’s; --speculative-num-stepsor--speculative-eagle-topkother than 1;- a TP size that does not divide the draft’s attention heads and MLP widths (TP 4 and TP 8 do).
temperature, top_k, top_p and min_p resolves to temperature: 0, top_k: 1. Setting any one of those fields is an explicit sampling choice, and the fields it leaves unset keep the checkpoint’s generation_config.json values, including temperature: 0.9: a request with only top_p: 1 samples at temperature 0.9 and is rejected. To combine sampling fields with DFly2, also send temperature: 0 or top_k: 1; otherwise the request fails with HTTP 400 and the message DFLY speculative decoding verifies greedily; send temperature=0 or top_k=1. DFly2 rejects non-greedy sampling, grammar-constrained decoding, frequency, presence or repetition penalties, a minimum token count, logit bias, custom logit processors, returned logprobs and returned hidden states.
DFly2 runs on CUDA only and needs the default flashmla_sparse DSA decode backend. Hierarchical KV cache is supported with DFly2.
Optional features
These are opt-in additions to the Deploy recipes.- Hierarchical KV cache (HiCache): enable it on either recipe in the Playground’s HiCache card, which adds
--enable-hierarchical-cache --hicache-ratio 2 --hicache-size 0 --hicache-write-policy write_through. Cached prefixes are written through to a host pool twice the size of each rank’s device KV pool, and a prefix evicted from the GPU reloads from host memory on its next hit. The host pool holds both the MLA latent KV and the DSA indexer keys, so budget host RAM for it on every rank. HiCache is refused with decode context parallelism. - DP-Attention (High-Throughput recipe): add
--attn-dp-sizewith a degree that divides TP (2, 4 or 8 at TP8; 2 or 4 at TP4). The server refuses DP-Attention with DFly2 and with prefill context parallelism. - Prefill context parallelism (High-Throughput recipe): add
--enable-prefill-cp --cp-strategy interleave --attn-cp-size 2for long prompts with text, images and video. The server refuses prefill CP with speculative decoding (including DFly2), DP-Attention, pipeline parallelism, PD disaggregation and--cp-strategy zigzag. - Decode context parallelism (High-Throughput recipe, B200/B300/GB300): on SM100, SM103 and SM107 GPUs,
--dcp-size 2stripes the MLA latent KV across each rank pair by position while the DSA indexer keys stay replicated, raising KV capacity at the cost of extra decode communication. Both DSA backends switch totrtllm, and the prefill CUDA graph is disabled. On H200 (SM90) and SM12x GPUs the server refuses--dcp-sizeat startup: the DCP merge needs the softmax LSE that only the TRT-LLM-gen sparse MLA kernel returns with learnable attention sinks, and that kernel runs only on SM100, SM103 and SM107. DCP is refused with DFly2, pipeline parallelism, DP-Attention, prefill CP, HiCache, the FP8 KV cache and PD disaggregation. See decode context parallelism. - FP8 KV cache (High-Throughput recipe, B200/B300/GB300): the KV cache is BF16 by default on every GPU. On SM100, SM103 and SM107,
--kv-cache-dtype fp8_e4m3stores it in FP8 and resolves both DSA backends totrtllm. The server refuses the FP8 KV cache on SM90 and SM12x, with decode or prefill context parallelism, and with DFly2.
Usage examples
The examples use the OpenAI Python client against either recipe on port 30000. They set no sampling fields, so the Low-Latency recipe decodes greedily and the High-Throughput recipe samples with the checkpoint’sgeneration_config.json defaults. The example outputs come from the H200 Low-Latency recipe; on the High-Throughput recipe the wording varies from run to run.
Chat
Thinking is on by default. To answer directly, passreasoning_effort: "no_think" through chat_template_kwargs:
Chat without thinking (Python)
Chat without thinking (Python)
Example
Example Output
Example Output
Output
Reasoning
The Deploy recipes pass--reasoning-parser auto, which resolves to the hunyuan parser. It returns the thinking in message.reasoning_content and the answer in message.content:
Reasoning (Python)
Reasoning (Python)
Example
Example Output
Example Output
Output
stream=True, the thinking arrives in delta.reasoning_content and the answer in delta.content:
Streaming with thinking (Python)
Streaming with thinking (Python)
Example
Images
Send images asimage_url content parts, as URLs or base64 data: URIs, in any position relative to the text. The image examples draw their test images locally with Pillow, so save this helper next to them as hy4_images.py:
Test image helper (Python)
Test image helper (Python)
hy4_images.py
Single image (Python)
Single image (Python)
Example
Example Output
Example Output
Output
Multiple images (Python)
Multiple images (Python)
Example
Example Output
Example Output
Output
Video
Send a clip as avideo_url content part. The server samples frames at 2 fps and adds a timestamp to each frame, so no client-side frame extraction is needed:
Video (Python)
Video (Python)
Example
Tool calling
The Deploy recipes pass--tool-call-parser auto, which resolves to the hunyuan parser and returns calls in message.tool_calls. The model thinks before calling, so print reasoning_content too:
Tool calling (Python)
Tool calling (Python)
Example
Example Output
Example Output
Output
tool_choice: "required", a named function and strict tools are enforced by a structural grammar, as are response_format and regex. Send these requests to the High-Throughput recipe: the Low-Latency recipe rejects them with HTTP 400 and the message DFLY speculative decoding does not support grammar decoding.
Notes and limitations
- Memory fraction: both strategies use the server’s automatic memory sizing. If you override memory settings for vision or long-context peaks, leave enough room for the KV pool and graph capture.
- DSA indexer fusion: on CUDA the DSA indexer runs fused by default, and
SGLANG_DISABLE_DSA_INDEXER_FUSION=1selects the unfused path. The two modes store indexer keys in different bases and the HiCache storage key does not distinguish them, so with a persistent L3 backend (for example--hicache-storage-backend file) use a fresh storage directory or namespace whenever the fusion mode changes, and never share one L3 namespace between fused and unfused workers. - Checkpoint config: the vision tower always runs unquantized, and the loader rejects quantized vision tensors and
tie_word_embeddings: true. Vendorsoft_logits_capping/soft_logits_capping_logitsmaps to the final logit softcap. - Video: there is no video accuracy benchmark for these recipes.
