ERNIE-5.5-Flash

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang

docker pull lmsysorg/sglang:dev-ernie-55-flash

• Low-Latency: ECHO speculative decoding (2 draft steps). Fastest replies for a single user or light load.
• High-Throughput: attention DP without a draft. Most tokens per second across many concurrent requests.

​

Playground

​

1. Model Introduction

​

2. Configuration Tips

• What gets set automatically. The model selects the ernie55 attention backend, --page-size 256, and --swa-full-tokens-ratio 0.1 on its own. --kv-cache-dtype defaults to fp8_e4m3; bfloat16 is the only other supported value.
• Parallelism. Attention TP (tp / attn-dp-size / attn-cp-size) must divide 8, the number of output groups. The high-throughput recipes use attention TP 1 (--attn-dp-size equal to --tp); the low-latency recipes use plain TP, except BF16 on H100, where TP 16 keeps --attn-dp-size 2 so attention TP stays at 8. Expert parallelism is optional: --ep N --moe-a2a-backend deepep routes experts over DeepEP, which needs --ep 4 or more (at --ep 2 each rank would hold 256 experts, past DeepEP’s per-rank limit). Prefill context parallelism (--attn-cp-size N --enable-prefill-cp --cp-strategy interleave) and pipeline parallelism also work; under PP use --moe-a2a-backend none.
• Long context. The released config allows 1,048,576 tokens. Pass --context-length 1048576, and lower --max-running-requests (to single digits on 2 × B300) and --cuda-graph-max-bs-decode: each rank’s KV pool must hold whole 1M-token requests, and a 1M prefill needs tens of GB of activation memory on top.
• DeepEP. When you add --moe-a2a-backend deepep, decode runs DeepEP low-latency inside a CUDA graph and each rank dispatches at most SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK tokens per step (default 128, at most 1024). Keep --cuda-graph-max-bs-decode × the tokens per request (1, or --speculative-num-steps + 1 with ECHO, or the block size + 1 with DSpark) at or below that value.

​

3. Advanced Usage

​

3.1 Completions (Base)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
resp = client.completions.create(
    model="baidu/ERNIE-5.5-Flash-Base",
    prompt="Question: What is 15% of 240?\nAnswer:",
    max_tokens=64,
    temperature=0,
    stop=["\n"],
)
print(resp.choices[0].text)

 36

​

3.2 Reasoning (ERNIE-5.5-Flash, coming soon)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
messages = [{"role": "user", "content": "What is 15% of 240?"}]

# Thinking is on by default.
msg = client.chat.completions.create(
    model="baidu/ERNIE-5.5-Flash", messages=messages
).choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Answer:", msg.content)

# Turn it off for one request.
msg = client.chat.completions.create(
    model="baidu/ERNIE-5.5-Flash",
    messages=messages,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
).choices[0].message
print("Answer:", msg.content)

​

3.3 Tool Calling (ERNIE-5.5-Flash, coming soon)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="baidu/ERNIE-5.5-Flash",
    messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
    tools=tools,
)
for call in resp.choices[0].message.tool_calls or []:
    print("Tool call:", call.function.name, call.function.arguments)

​

3.4 HiCache (Hierarchical KV Caching)

​

3.5 Speculative Decoding

sglang serve --model-path baidu/ERNIE-5.5-Flash \
  --tp 2 --context-length 131072 --max-running-requests 256 \
  --reasoning-parser ernie55 --tool-call-parser ernie55 \
  --speculative-algorithm DSPARK --enable-dp-lm-head \
  --host 0.0.0.0 --port 30000

​

3.6 PD Disaggregation