> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sglang.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Hy4

> Deploy Tencent Hy4 with SGLang — a 770B-parameter MoE vision-language model with DeepSeek Sparse Attention and DFly2 speculative decoding, in FP8 on H200, B200, B300 and GB300.

## Deployment

> [accordion] Install SGLang
>
> Start an interactive container with GPU access, shared memory, the Hugging Face cache, port 30000, and your checkpoint token. Replace `<your-hf-token>` before running this command:
>
> ```bash Command theme={null}
> export HF_TOKEN=<your-hf-token>
> docker run --gpus all -it \
>   --shm-size 32g \
>   -p 30000:30000 \
>   -v ~/.cache/huggingface:/root/.cache/huggingface \
>   --env HF_TOKEN \
>   --ipc=host \
>   lmsysorg/sglang:hy4 \
>   bash
> ```
>
> At the resulting container shell, run the **Python** panel's `sglang serve` command. Alternatively, run the **Docker** panel's complete `docker run` command directly on the host. Keep the server running while you send requests to port 30000.

Pick your hardware and strategy to generate the launch command. Both strategies serve the FP8 checkpoint at TP8 on 8× H200/B200 or TP4 on 4× B300/GB300 and use automatic memory sizing:

* **Low-Latency** — DFly2 speculative decoding with the draft layers bundled in the checkpoint. Pick it for interactive traffic such as single-user chat and agents that decode greedily; see [DFly2 requirements](#dfly2-requirements) for the request restrictions.
* **High-Throughput** — no speculative decoding. Pick it for multi-user and batch traffic, for sampled decoding and grammar-constrained tool calls, and as the base for DP-Attention, context parallelism and the FP8 KV cache.

## Checkpoints

<table>
  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Weights</th>
      <th>Recipes on this page</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tencent/Hy4-FP8`</td>
      <td>\~786 GB, block FP8</td>
      <td>8× H200/B200 at TP8, 4× B300/GB300 at TP4</td>
    </tr>

    <tr>
      <td>`tencent/Hy4`</td>
      <td>\~1.55 TB, BF16</td>
      <td>None: the weights exceed TP8 on H200/B200 and TP4 on B300/GB300</td>
    </tr>
  </tbody>
</table>

`tencent/Hy4-FP8` stores the routed experts, the attention output and gate projections, and most of the DFly2 layers' linear projections in block FP8: e4m3 weights in 128×128 blocks with dynamically quantized group-128 activations. The remaining attention projections, the DSA indexer, the first layer's attention and dense MLP, the shared experts, `lm_head` and the vision tower stay in BF16. SGLang reads the quantization from the checkpoint, so the recipes pass no `--quantization` flag.

Both checkpoints declare a 1,048,576-token context. No recipe pins `--context-length`, so the server uses the declared length; the KV pool, not the declared length, bounds how many tokens fit at once.

## Model capabilities

Hy4 pairs a 770B-parameter language backbone (MLA + DSA sparse attention + 256-expert MoE + iHC) with a 40-layer vision transformer and a Conv2d spatial-merge adapter, for image and video understanding with dynamic resolution (32-pixel alignment, 512²–2048² budget) and 3-channel mRoPE position encoding. The 770B count excludes the ViT, the adapter and the DFly2 layers.

* **Images**: single and multiple images per request, in any position relative to text. Both the OpenAI chat API (`image_url` parts) and raw prompts with `<image>` placeholders work.
* **Video**: sampled-frame video input (fps=2) with per-frame timestamp tokens. Long or high-resolution clips can exhaust the KV pool, so leave headroom for long videos.
* **Tool calling + reasoning**: `--reasoning-parser auto --tool-call-parser auto` resolve to the `hunyuan` parsers. `tool_choice: "required"`, a named function, and `"auto"` with `strict` tools are enforced by a native structural grammar: the call envelope, tool names and argument keys are constrained, while argument values are free text and are not validated against the schema. A named function or `parallel_tool_calls: false` allows exactly one call. DFly2 rejects these grammar-constrained requests; serve them from the High-Throughput recipe.
* **DFly2 speculative decoding**: the checkpoint bundles five DFly2 draft layers. One draft forward runs the bonus token, mixed with the target's final hidden state, plus six mask tokens, and one target pass verifies the seven-token block. Each draft layer keeps its KV in the pool of a target layer and reuses a target indexer's sparse-attention top-k, so the draft builds no indexer and no KV pool of its own. The seven-token block comes from the checkpoint's `dfly_block_size`, so the recipes pass no block-size flag. With the server defaults, the decode CUDA graph resolves to `full` and captures the draft forward together with the target verify, the target prefill replays through a `breakable` CUDA graph, and overlap scheduling stays on.

## Playground

The Playground adds knobs on top of the selected Deploy cell: DP-Attention and prefill context parallelism (CP2) on the High-Throughput recipe, the hierarchical KV cache, and the speculative preset. Everything else is inherited from the base recipe.

## DFly2 requirements

The server validates DFly2 before loading weights and refuses to start with any of:

* a decode CUDA graph backend other than `full` or `disabled`, a prefill CUDA graph backend other than `breakable` or `disabled`, two-batch overlap or mixed chunked prefill;
* DP-Attention, pipeline parallelism, prefill context parallelism (`--attn-cp-size` above 1), decode context parallelism (`--dcp-size` above 1) or PD disaggregation;
* a model dtype other than BF16, or a KV cache dtype other than BF16;
* LoRA;
* a separate draft checkpoint (`--speculative-draft-model-path`), a draft attention backend, or a draft quantization or KV cache dtype that differs from the target's;
* `--speculative-num-steps` or `--speculative-eagle-topk` other than 1;
* a TP size that does not divide the draft's attention heads and MLP widths (TP 4 and TP 8 do).

DFly2 verifies drafts greedily. A request that sets none of `temperature`, `top_k`, `top_p` and `min_p` resolves to `temperature: 0`, `top_k: 1`. Setting any one of those fields is an explicit sampling choice, and the fields it leaves unset keep the checkpoint's `generation_config.json` values, including `temperature: 0.9`: a request with only `top_p: 1` samples at temperature 0.9 and is rejected. To combine sampling fields with DFly2, also send `temperature: 0` or `top_k: 1`; otherwise the request fails with HTTP 400 and the message `DFLY speculative decoding verifies greedily; send temperature=0 or top_k=1.` DFly2 rejects non-greedy sampling, grammar-constrained decoding, frequency, presence or repetition penalties, a minimum token count, logit bias, custom logit processors, returned logprobs and returned hidden states.

DFly2 runs on CUDA only and needs the default `flashmla_sparse` DSA decode backend. Hierarchical KV cache is supported with DFly2.

## Optional features

These are opt-in additions to the Deploy recipes.

* **Hierarchical KV cache (HiCache)**: enable it on either recipe in the Playground's **HiCache** card, which adds `--enable-hierarchical-cache --hicache-ratio 2 --hicache-size 0 --hicache-write-policy write_through`. Cached prefixes are written through to a host pool twice the size of each rank's device KV pool, and a prefix evicted from the GPU reloads from host memory on its next hit. The host pool holds both the MLA latent KV and the DSA indexer keys, so budget host RAM for it on every rank. HiCache is refused with decode context parallelism.
* **DP-Attention** (High-Throughput recipe): add `--attn-dp-size` with a degree that divides TP (2, 4 or 8 at TP8; 2 or 4 at TP4). The server refuses DP-Attention with DFly2 and with prefill context parallelism.
* **Prefill context parallelism** (High-Throughput recipe): add `--enable-prefill-cp --cp-strategy interleave --attn-cp-size 2` for long prompts with text, images and video. The server refuses prefill CP with speculative decoding (including DFly2), DP-Attention, pipeline parallelism, PD disaggregation and `--cp-strategy zigzag`.
* **Decode context parallelism** (High-Throughput recipe, B200/B300/GB300): on SM100, SM103 and SM107 GPUs, `--dcp-size 2` stripes the MLA latent KV across each rank pair by position while the DSA indexer keys stay replicated, raising KV capacity at the cost of extra decode communication. Both DSA backends switch to `trtllm`, and the prefill CUDA graph is disabled. On H200 (SM90) and SM12x GPUs the server refuses `--dcp-size` at startup: the DCP merge needs the softmax LSE that only the TRT-LLM-gen sparse MLA kernel returns with learnable attention sinks, and that kernel runs only on SM100, SM103 and SM107. DCP is refused with DFly2, pipeline parallelism, DP-Attention, prefill CP, HiCache, the FP8 KV cache and PD disaggregation. See [decode context parallelism](https://docs.sglang.io/docs/advanced_features/dcp.md).
* **FP8 KV cache** (High-Throughput recipe, B200/B300/GB300): the KV cache is BF16 by default on every GPU. On SM100, SM103 and SM107, `--kv-cache-dtype fp8_e4m3` stores it in FP8 and resolves both DSA backends to `trtllm`. The server refuses the FP8 KV cache on SM90 and SM12x, with decode or prefill context parallelism, and with DFly2.

## Usage examples

The examples use the OpenAI Python client against either recipe on port 30000. They set no sampling fields, so the Low-Latency recipe decodes greedily and the High-Throughput recipe samples with the checkpoint's `generation_config.json` defaults. The example outputs come from the H200 Low-Latency recipe; on the High-Throughput recipe the wording varies from run to run.

### Chat

Thinking is on by default. To answer directly, pass `reasoning_effort: "no_think"` through `chat_template_kwargs`:

> [accordion] Chat without thinking (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{"role": "user", "content": "Hi"}],
>     max_tokens=32,
>     extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
> )
>
> print(response.choices[0].message.content)
> ```

> [accordion] Example Output
>
> ```text Output theme={null}
> Hello! How can I help you today? 😊
> ```

### Reasoning

The Deploy recipes pass `--reasoning-parser auto`, which resolves to the `hunyuan` parser. It returns the thinking in `message.reasoning_content` and the answer in `message.content`:

> [accordion] Reasoning (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{
>         "role": "user",
>         "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Put the final answer after '####'.",
>     }],
>     max_tokens=2048,
> )
>
> message = response.choices[0].message
> print("=============== Thinking =================")
> print(message.reasoning_content)
> print("=============== Content =================")
> print(message.content)
> ```

> [accordion] Example Output
>
> ```text Output theme={null}
> =============== Thinking =================
> We need answer simple word problem. Need reason. Need final with answer after '####'. Need be careful: "sold clips to 48 of her friends in April" Ambiguity: sold clips to 48 friends, not necessarily 48 clips? Usually "sold clips to 48 of her friends" means sold 48 clips? Need parse. Then "sold half as many clips in May" means half as many clips as in April. Need total. If April = 48 clips, May = 24, total = 72. Need final answer after '####'. Could include concise explanation. Need ensure final answer after '####'. Maybe "April: 48 clips. May: half of 48 = 24. Total = 48 + 24 = 72. #### 72". Need maybe "#### 72" final answer after marker. Need not overcomplicate. Need consider if "to 48 of her friends" could mean 48 friends, each bought one clip? In elementary word problems, "sold clips to 48 of her friends" likely means 48 clips. Need answer 72. Need final.
> =============== Content =================
> April: 48 clips
> May: half of 48 = 24 clips
> Total: 48 + 24 = 72 clips
>
> #### 72
> ```

With `stream=True`, the thinking arrives in `delta.reasoning_content` and the answer in `delta.content`:

> [accordion] Streaming with thinking (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> stream = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{
>         "role": "user",
>         "content": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Put the final answer after '####'.",
>     }],
>     max_tokens=2048,
>     stream=True,
> )
>
> thinking_started = False
> answer_started = False
> for chunk in stream:
>     if not chunk.choices:
>         continue
>     delta = chunk.choices[0].delta
>     if getattr(delta, "reasoning_content", None):
>         if not thinking_started:
>             print("=============== Thinking =================", flush=True)
>             thinking_started = True
>         print(delta.reasoning_content, end="", flush=True)
>     if delta.content:
>         if not answer_started:
>             print("\n=============== Content =================", flush=True)
>             answer_started = True
>         print(delta.content, end="", flush=True)
> print()
> ```

### Images

Send images as `image_url` content parts, as URLs or base64 `data:` URIs, in any position relative to the text. The image examples draw their test images locally with Pillow, so save this helper next to them as `hy4_images.py`:

> [accordion] Test image helper (Python)
>
> ```python hy4_images.py theme={null}
> import base64
> import io
>
> from PIL import Image, ImageDraw, ImageFont
>
>
> def make_image(shape, color, label):
>     image = Image.new("RGB", (448, 448), "white")
>     draw = ImageDraw.Draw(image)
>     box = (100, 60, 348, 308)
>     if shape == "circle":
>         draw.ellipse(box, fill=color)
>     elif shape == "square":
>         draw.rectangle(box, fill=color)
>     else:
>         draw.polygon([(224, 60), (348, 308), (100, 308)], fill=color)
>     draw.text((110, 350), label, fill="black", font=ImageFont.truetype("DejaVuSans-Bold.ttf", 40))
>     buffer = io.BytesIO()
>     image.save(buffer, "PNG")
>     return "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode()
> ```

> [accordion] Single image (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> from hy4_images import make_image
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url", "image_url": {"url": make_image("circle", "red", "RED 7231")}},
>             {"type": "text", "text": "Describe the shape, its color and the text in the image."},
>         ],
>     }],
>     max_tokens=128,
>     extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
> )
>
> print(response.choices[0].message.content)
> ```

> [accordion] Example Output
>
> ```text Output theme={null}
> The shape in the image is a **circle** (or a round disc). Its color is a bright, vivid **red** (specifically labeled as "RED 7231"). Below the red circle, the text reads **"RED 7231"** in bold, black, uppercase letters (with a space between "RED" and "7231").
> ```

> [accordion] Multiple images (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> from hy4_images import make_image
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url", "image_url": {"url": make_image("circle", "red", "RED 7231")}},
>             {"type": "image_url", "image_url": {"url": make_image("triangle", "green", "GREEN 5582")}},
>             {"type": "text", "text": "There are two images. For each image, in order, give the text written under it, one per line as 'Image k: <text>'."},
>         ],
>     }],
>     max_tokens=128,
>     extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
> )
>
> print(response.choices[0].message.content)
> ```

> [accordion] Example Output
>
> ```text Output theme={null}
> Image 1: RED 7231
> Image 2: GREEN 5582
> ```

### Video

Send a clip as a `video_url` content part. The server samples frames at 2 fps and adds a timestamp to each frame, so no client-side frame extraction is needed:

> [accordion] Video (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "video_url", "video_url": {"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"}},
>             {"type": "text", "text": "What is happening in this video? Answer in two sentences."},
>         ],
>     }],
>     max_tokens=2048,
> )
>
> message = response.choices[0].message
> print("Reasoning:", message.reasoning_content)
> print("Content:", message.content)
> ```

### Tool calling

The Deploy recipes pass `--tool-call-parser auto`, which resolves to the `hunyuan` parser and returns calls in `message.tool_calls`. The model thinks before calling, so print `reasoning_content` too:

> [accordion] Tool calling (Python)
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string", "description": "City name"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-FP8",
>     messages=[{"role": "user", "content": "What's the weather like in Paris right now? Use the tool."}],
>     tools=tools,
>     tool_choice="auto",
>     max_tokens=1024,
> )
>
> message = response.choices[0].message
> print("Reasoning:", message.reasoning_content)
> print("Content:", message.content)
> for call in message.tool_calls or []:
>     print("Tool call:", call.function.name, call.function.arguments)
> print("Finish reason:", response.choices[0].finish_reason)
> ```

> [accordion] Example Output
>
> ```text Output theme={null}
> Reasoning: The user wants to know the weather in Paris right now and explicitly asks to use the tool. I need to call the get_weather function with city="Paris". Let me do that.
> Content: I'll check the current weather in Paris for you.
> Tool call: get_weather {"city": "Paris"}
> Finish reason: tool_calls
> ```

`tool_choice: "required"`, a named function and `strict` tools are enforced by a structural grammar, as are `response_format` and `regex`. Send these requests to the High-Throughput recipe: the Low-Latency recipe rejects them with HTTP 400 and the message `DFLY speculative decoding does not support grammar decoding.`

## Notes and limitations

* **Memory fraction**: both strategies use the server's automatic memory sizing. If you override memory settings for vision or long-context peaks, leave enough room for the KV pool and graph capture.
* **DSA indexer fusion**: on CUDA the DSA indexer runs fused by default, and `SGLANG_DISABLE_DSA_INDEXER_FUSION=1` selects the unfused path. The two modes store indexer keys in different bases and the HiCache storage key does not distinguish them, so with a persistent L3 backend (for example `--hicache-storage-backend file`) use a fresh storage directory or namespace whenever the fusion mode changes, and never share one L3 namespace between fused and unfused workers.
* **Checkpoint config**: the vision tower always runs unquantized, and the loader rejects quantized vision tensors and `tie_word_embeddings: true`. Vendor `soft_logits_capping` / `soft_logits_capping_logits` maps to the final logit softcap.
* **Video**: there is no video accuracy benchmark for these recipes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.