Skip to main content

​
Deployment

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.
pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

# Install from source (main carries the suffix-aware `hunyuan` parser + the
# HYV3 model code). Once a tagged release picks it up, `uv pip install sglang`
# is enough.
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv pip install -e python
Run the Python output of the command panel below in that environment.
Pick your hardware + recipe to generate the launch command.
  • Low-Latency — fastest reply for a single user. Pick for chat.
  • Balanced — good speed with several users at once. Use for typical multi-user serving.

Panel controls (top of the command box):

  • Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
  • ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
  • $ cURL — a sample request against localhost:30000 to confirm the server is up.
  • ⚙ Env — edits the placeholders (HOST_IP, PORT, HF_TOKEN, NODE_RANK, NODE0_IP) the command and cURL share. Persists in localStorage across cookbooks.
  • Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.

​
Playground

The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change. The knobs come in two flavors:
  • Built-in SGLang features — parallelism overrides (TP / CP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
  • Hy3 specific — --tool-call-parser auto / --reasoning-parser auto (auto-detect Hy3’s suffix-aware hunyuan parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).
Lines highlighted green are added by your overrides; lines with red strikethrough were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base’s Verified badge; any actual change flips it to Not Verified until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · $ cURL · ⚙ Env from the Deploy panel, plus one extra:

  • Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you’ve actually run the command on your hardware and confirmed it works.

​
1. Model Introduction

Hy3 is Tencent’s third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving. Key Features:
  • MoE Architecture: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
  • Hybrid Thinking: Reasoning modes (high, low, no_think) controllable via OpenAI-standard reasoning_effort, allowing the same weights to trade off latency and depth of reasoning.
  • Native Tool Calling: Trained on a structured grammar. Pairs with SGLang’s hunyuan tool-call parser for streaming OpenAI-compatible function-calling output.
  • Long Context: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
  • Multi-Token Prediction (MTP): Ships with a built-in MTP draft module enabling speculative decoding out of the box.
Available Model: Recommended Generation Parameters:
ParameterValue
temperature0.9
top_p1.0
reasoning_efforthigh / low (thinking) or no_think (instant)
Special tokens. The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. <tool_calls:TAG> instead of the bare <tool_calls>). SGLang’s hunyuan parsers resolve the real token strings from the tokenizer vocab at runtime (PR #29920), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why --reasoning-parser hunyuan / --tool-call-parser hunyuan work out of the box on the shipping model.

​
2. Configuration Tips

Hardware requirements (BF16, ~590GB weights):
GPUVRAMTPNotes
H200141GB8minimum single-node for BF16
B200192GB4BF16 590GB → 148GB/GPU
B300 / GB300288GB4BF16 590GB → 148GB/GPU; ample KV headroom
GB200192GB4single-node 4×192GB = 768GB fits BF16 590GB
Blackwell attention backend. On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the trtllm_mha attention backend for HYV3’s MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason. MTP (Multi-Token Prediction, EAGLE).
  • low-latency: steps=3, draft-tokens=4 → largest win at bs=1.
  • balanced: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.
reasoning_effort vs thinking. The Hy3 chat template is driven by reasoning_effort (high / low / no_think), NOT by the thinking flag that some other families use. The default is no_think (instant). To opt into thinking, pass reasoning_effort="high" on the request (the OpenAI-standard field; sglang forwards it to the template). reasoning_effort: max is rejected by sglang — use high. For eval, sgl-eval’s --thinking flag translates to reasoning_effort="high" for Hy3, so the benchmark commands below use it as-is.

​
3. Advanced Usage

​
3.1 Reasoning (Hybrid Thinking)

Hy3 is a hybrid-thinking model. Control the thinking budget via reasoning_effort:
  • high / low — increasing amounts of chain-of-thought in reasoning_content
  • no_think — skip thinking entirely (instant responses, content-only)
Enable the reasoning parser during deployment so the thinking section is separated into reasoning_content:
Command
sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto
Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)
Output
=============== Thinking =================
We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
=============== Content =================
To find 15% of 240, follow these steps:

1. 15% = 15/100 or 0.15.
2. Multiply 240 by 0.15: 0.15 × 240 = 36.
3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.

Thus, 15% of 240 is 36.
Example
response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="no_think",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)
Output
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.

​
3.2 Tool Calling

Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:
Command
sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto
Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")
Output
Reasoning: None
Content:   I'll get the current weather for Beijing in Fahrenheit for you.
Tool Call: get_weather
  Arguments: {"city": "Beijing", "unit": "fahrenheit"}
Example
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
    stream=True,
)

tool_buffer = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
    for tc in delta.tool_calls or []:
        buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function and tc.function.name:
            buf["name"] += tc.function.name
        if tc.function and tc.function.arguments:
            buf["args"] += tc.function.arguments

for idx, buf in tool_buffer.items():
    print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
Output
I'll check the current weather in Beijing for you using Fahrenheit.
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})