Skip to main content

Deployment

Start an interactive container with GPU access, shared memory, the Hugging Face cache, port 30000, and your checkpoint token. Replace <your-hf-token> before running this command:
Command
At the resulting container shell, run the Python panel’s sglang serve command. Alternatively, run the Docker panel’s complete docker run command directly on the host. Keep the server running while you send requests to port 30000.
Pick your hardware and strategy to generate the launch command. Both strategies serve the FP8 checkpoint at TP8 on 8× H200/B200 or TP4 on 4× B300/GB300 and use automatic memory sizing:
  • Low-Latency — DFly2 speculative decoding with the draft layers bundled in the checkpoint. Pick it for interactive traffic such as single-user chat and agents that decode greedily; see DFly2 requirements for the request restrictions.
  • High-Throughput — no speculative decoding. Pick it for multi-user and batch traffic, for sampled decoding and grammar-constrained tool calls, and as the base for DP-Attention, context parallelism and the FP8 KV cache.

Checkpoints

CheckpointWeightsRecipes on this page
tencent/Hy4-FP8~786 GB, block FP88× H200/B200 at TP8, 4× B300/GB300 at TP4
tencent/Hy4~1.55 TB, BF16None: the weights exceed TP8 on H200/B200 and TP4 on B300/GB300
tencent/Hy4-FP8 stores the routed experts, the attention output and gate projections, and most of the DFly2 layers’ linear projections in block FP8: e4m3 weights in 128×128 blocks with dynamically quantized group-128 activations. The remaining attention projections, the DSA indexer, the first layer’s attention and dense MLP, the shared experts, lm_head and the vision tower stay in BF16. SGLang reads the quantization from the checkpoint, so the recipes pass no --quantization flag. Both checkpoints declare a 1,048,576-token context. No recipe pins --context-length, so the server uses the declared length; the KV pool, not the declared length, bounds how many tokens fit at once.

Model capabilities

Hy4 pairs a 770B-parameter language backbone (MLA + DSA sparse attention + 256-expert MoE + iHC) with a 40-layer vision transformer and a Conv2d spatial-merge adapter, for image and video understanding with dynamic resolution (32-pixel alignment, 512²–2048² budget) and 3-channel mRoPE position encoding. The 770B count excludes the ViT, the adapter and the DFly2 layers.
  • Images: single and multiple images per request, in any position relative to text. Both the OpenAI chat API (image_url parts) and raw prompts with <image> placeholders work.
  • Video: sampled-frame video input (fps=2) with per-frame timestamp tokens. Long or high-resolution clips can exhaust the KV pool, so leave headroom for long videos.
  • Tool calling + reasoning: --reasoning-parser auto --tool-call-parser auto resolve to the hunyuan parsers. tool_choice: "required", a named function, and "auto" with strict tools are enforced by a native structural grammar: the call envelope, tool names and argument keys are constrained, while argument values are free text and are not validated against the schema. A named function or parallel_tool_calls: false allows exactly one call. DFly2 rejects these grammar-constrained requests; serve them from the High-Throughput recipe.
  • DFly2 speculative decoding: the checkpoint bundles five DFly2 draft layers. One draft forward runs the bonus token, mixed with the target’s final hidden state, plus six mask tokens, and one target pass verifies the seven-token block. Each draft layer keeps its KV in the pool of a target layer and reuses a target indexer’s sparse-attention top-k, so the draft builds no indexer and no KV pool of its own. The seven-token block comes from the checkpoint’s dfly_block_size, so the recipes pass no block-size flag. With the server defaults, the decode CUDA graph resolves to full and captures the draft forward together with the target verify, the target prefill replays through a breakable CUDA graph, and overlap scheduling stays on.

Playground

The Playground adds knobs on top of the selected Deploy cell: DP-Attention and prefill context parallelism (CP2) on the High-Throughput recipe, the hierarchical KV cache, and the speculative preset. Everything else is inherited from the base recipe.

DFly2 requirements

The server validates DFly2 before loading weights and refuses to start with any of:
  • a decode CUDA graph backend other than full or disabled, a prefill CUDA graph backend other than breakable or disabled, two-batch overlap or mixed chunked prefill;
  • DP-Attention, pipeline parallelism, prefill context parallelism (--attn-cp-size above 1), decode context parallelism (--dcp-size above 1) or PD disaggregation;
  • a model dtype other than BF16, or a KV cache dtype other than BF16;
  • LoRA;
  • a separate draft checkpoint (--speculative-draft-model-path), a draft attention backend, or a draft quantization or KV cache dtype that differs from the target’s;
  • --speculative-num-steps or --speculative-eagle-topk other than 1;
  • a TP size that does not divide the draft’s attention heads and MLP widths (TP 4 and TP 8 do).
DFly2 verifies drafts greedily. A request that sets none of temperature, top_k, top_p and min_p resolves to temperature: 0, top_k: 1. Setting any one of those fields is an explicit sampling choice, and the fields it leaves unset keep the checkpoint’s generation_config.json values, including temperature: 0.9: a request with only top_p: 1 samples at temperature 0.9 and is rejected. To combine sampling fields with DFly2, also send temperature: 0 or top_k: 1; otherwise the request fails with HTTP 400 and the message DFLY speculative decoding verifies greedily; send temperature=0 or top_k=1. DFly2 rejects non-greedy sampling, grammar-constrained decoding, frequency, presence or repetition penalties, a minimum token count, logit bias, custom logit processors, returned logprobs and returned hidden states. DFly2 runs on CUDA only and needs the default flashmla_sparse DSA decode backend. Hierarchical KV cache is supported with DFly2.

Optional features

These are opt-in additions to the Deploy recipes.
  • Hierarchical KV cache (HiCache): enable it on either recipe in the Playground’s HiCache card, which adds --enable-hierarchical-cache --hicache-ratio 2 --hicache-size 0 --hicache-write-policy write_through. Cached prefixes are written through to a host pool twice the size of each rank’s device KV pool, and a prefix evicted from the GPU reloads from host memory on its next hit. The host pool holds both the MLA latent KV and the DSA indexer keys, so budget host RAM for it on every rank. HiCache is refused with decode context parallelism.
  • DP-Attention (High-Throughput recipe): add --attn-dp-size with a degree that divides TP (2, 4 or 8 at TP8; 2 or 4 at TP4). The server refuses DP-Attention with DFly2 and with prefill context parallelism.
  • Prefill context parallelism (High-Throughput recipe): add --enable-prefill-cp --cp-strategy interleave --attn-cp-size 2 for long prompts with text, images and video. The server refuses prefill CP with speculative decoding (including DFly2), DP-Attention, pipeline parallelism, PD disaggregation and --cp-strategy zigzag.
  • Decode context parallelism (High-Throughput recipe, B200/B300/GB300): on SM100, SM103 and SM107 GPUs, --dcp-size 2 stripes the MLA latent KV across each rank pair by position while the DSA indexer keys stay replicated, raising KV capacity at the cost of extra decode communication. Both DSA backends switch to trtllm, and the prefill CUDA graph is disabled. On H200 (SM90) and SM12x GPUs the server refuses --dcp-size at startup: the DCP merge needs the softmax LSE that only the TRT-LLM-gen sparse MLA kernel returns with learnable attention sinks, and that kernel runs only on SM100, SM103 and SM107. DCP is refused with DFly2, pipeline parallelism, DP-Attention, prefill CP, HiCache, the FP8 KV cache and PD disaggregation. See decode context parallelism.
  • FP8 KV cache (High-Throughput recipe, B200/B300/GB300): the KV cache is BF16 by default on every GPU. On SM100, SM103 and SM107, --kv-cache-dtype fp8_e4m3 stores it in FP8 and resolves both DSA backends to trtllm. The server refuses the FP8 KV cache on SM90 and SM12x, with decode or prefill context parallelism, and with DFly2.

Usage examples

The examples use the OpenAI Python client against either recipe on port 30000. They set no sampling fields, so the Low-Latency recipe decodes greedily and the High-Throughput recipe samples with the checkpoint’s generation_config.json defaults. The example outputs come from the H200 Low-Latency recipe; on the High-Throughput recipe the wording varies from run to run.

Chat

Thinking is on by default. To answer directly, pass reasoning_effort: "no_think" through chat_template_kwargs:
Example
Output

Reasoning

The Deploy recipes pass --reasoning-parser auto, which resolves to the hunyuan parser. It returns the thinking in message.reasoning_content and the answer in message.content:
Example
Output
With stream=True, the thinking arrives in delta.reasoning_content and the answer in delta.content:
Example

Images

Send images as image_url content parts, as URLs or base64 data: URIs, in any position relative to the text. The image examples draw their test images locally with Pillow, so save this helper next to them as hy4_images.py:
hy4_images.py
Example
Output
Example
Output

Video

Send a clip as a video_url content part. The server samples frames at 2 fps and adds a timestamp to each frame, so no client-side frame extraction is needed:
Example

Tool calling

The Deploy recipes pass --tool-call-parser auto, which resolves to the hunyuan parser and returns calls in message.tool_calls. The model thinks before calling, so print reasoning_content too:
Example
Output
tool_choice: "required", a named function and strict tools are enforced by a structural grammar, as are response_format and regex. Send these requests to the High-Throughput recipe: the Low-Latency recipe rejects them with HTTP 400 and the message DFLY speculative decoding does not support grammar decoding.

Notes and limitations

  • Memory fraction: both strategies use the server’s automatic memory sizing. If you override memory settings for vision or long-context peaks, leave enough room for the KV pool and graph capture.
  • DSA indexer fusion: on CUDA the DSA indexer runs fused by default, and SGLANG_DISABLE_DSA_INDEXER_FUSION=1 selects the unfused path. The two modes store indexer keys in different bases and the HiCache storage key does not distinguish them, so with a persistent L3 backend (for example --hicache-storage-backend file) use a fresh storage directory or namespace whenever the fusion mode changes, and never share one L3 namespace between fused and unfused workers.
  • Checkpoint config: the vision tower always runs unquantized, and the loader rejects quantized vision tensors and tie_word_embeddings: true. Vendor soft_logits_capping / soft_logits_capping_logits maps to the final logit softcap.
  • Video: there is no video accuracy benchmark for these recipes.