Hy4-Preview

​

Deployment

docker pull lmsysorg/sglang:hy4-preview

• Low-Latency — MTP speculative decoding on (steps=3, draft-tokens=4). Fastest reply for a single user; pick for chat.
• High-Throughput — MTP off. At saturation the draft+verify overhead outweighs the speedup; best for batch jobs.

​

Playground

• Built-in SGLang features — parallelism overrides (TP / DP-Attention), the MoE a2a backend (DeepEP, which sets EP = TP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
• Hy4 specific — --reasoning-parser auto / --tool-call-parser auto resolve to the suffix-aware Hunyuan parsers and read the real structural-token strings from the tokenizer vocab at runtime.

​

1. Model Introduction

• MoE: 78 layers — layer 0 is a dense MLP, the remaining 77 are sparse MoE with 256 routed experts + 1 shared expert, top-8 sigmoid-scored routing (routed scaling 2.827), expert intermediate size 2048, and bounded SwiGLU (clamp 10.0).
• MLA + DSA on every layer: Multi-head Latent Attention (q_lora_rank 2048, kv_lora_rank 512, 192 nope + 64 rope head dims, v_head_dim 256) under DeepSeek Sparse Attention — indexer top-k 2048 with 32 index heads, indexers alternating full/shared (one full indexer every 4 layers), and an FP8 index cache.
• iHC residual control: Hunyuan’s own inter-layer residual gating (enable_ihc, hc_mult 4) with pre- and post-residual gate groups. Semantically distinct from DeepSeek-V4’s mHC (no combination step, no Sinkhorn) — the implementations are not interchangeable.
• Gated MLA + attention sink: element-wise attention output gating evaluated in fp32, plus a learnable per-head attention sink propagated through all attention backends.
• MTP (NextN): one built-in multi-token-prediction draft layer (model.mtp_layers.0, 10B parameters with ~0.7B active, present in both checkpoints) enabling speculative decoding out of the box.
• Long context: 1M max positions (RoPE theta 1e7). Size --context-length to your KV budget — the sizing table in §2 suggests 262,144 (131,072 on H200).

• tencent/Hy4-preview — BF16 (~1.5TB weights)
• tencent/Hy4-preview-FP8 — MXFP8 (ModelOpt recipe, UE8M0 group-32 weight scales, dynamic activations; ~760GB weights)

​

2. Configuration Tips

​

3. Advanced Usage

​

3.1 Reasoning (reasoning_effort)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy4-preview-FP8",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)

Pending update — will be captured verbatim from a live Hy4-Preview server.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy4-preview-FP8",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)

Pending update — will be captured verbatim from a live Hy4-Preview server.

​

3.2 Tool Calling

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy4-preview-FP8",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")

Pending update — will be captured verbatim from a live Hy4-Preview server.