Speculative Decoding

SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.

Summary

Jump to sections

EAGLE Decoding
Multi Token Prediction
UNO decoding
DFlash Decoding
Standalone Speculative Decoding (Small Draft Model)
Speculative Decoding V2 (Overlap Scheduler)
Ngram Speculative Decoding
Full Parameter Reference
OOM Troubleshooting
References

Quick guidance

Best speed/quality (recommended): Use EAGLE-3 with --speculative-algorithm EAGLE3.
Strong default / broad compatibility: Use EAGLE-2 with --speculative-algorithm EAGLE.
Workload acceptance changes over time: Use Adaptive speculative decoding on top of EAGLE with --speculative-eagle-topk 1.
Lower lm_head overhead for EAGLE-2: Enable FR-Spec with --speculative-token-map.
Model is MTP-enabled: Use MTP via speculative decoding (often with small speculative_num_steps/topk/num_draft_tokens, see the example section).
You have a UNO adapter for the target model: Use UNO with --speculative-algorithm UNO and --uno-lora-path ....
You have a DFlash draft checkpoint: Use DFLASH with --speculative-algorithm DFLASH and --speculative-draft-model-path ....
You have a smaller draft LLM: Use STANDALONE (--speculative-algorithm STANDALONE).
No extra model available: Use NGRAM (--speculative-algorithm NGRAM, CUDA-only).

Method comparison (mini table)

Method	Draft source	Separate draft model?	How to enable	Notes / constraints
EAGLE-2	EAGLE draft model (feature drafting + tree)	Typically yes	`—speculative-algorithm EAGLE` + `—speculative-draft-model-path …`	Tune `—speculative-num-steps`, `—speculative-eagle-topk`, `—speculative-num-draft-tokens`
EAGLE-2 + `torch.compile`	Same as EAGLE-2	Typically yes	Add `—enable-torch-compile` (optionally `—torch-compile-max-bs`)	Benefit varies by hardware/model; benchmark to verify
EAGLE-2 + FR-Spec	Same as EAGLE-2 + token subset	Typically yes	Add `—speculative-token-map …`	Reduces `lm_head` overhead with high-frequency token vocab
EAGLE-3	EAGLE3 draft model	Yes	`—speculative-algorithm EAGLE3` + `—speculative-draft-model-path …`	Best throughput in the benchmark below
MTP	Built-in multi-token heads (model-specific)	Often no	See Multi Token Prediction section	Uses speculative workflow; draft path may be auto-handled for some models
UNO	Target model with a trained UNO LoRA (linear chain or tree)	No	`—speculative-algorithm UNO` + `—uno-lora-path …`	CUDA and FA3; currently requires TP=PP=1
DFLASH	DFlash draft model (linear block verification)	Yes	`—speculative-algorithm DFLASH` + `—speculative-draft-model-path …`	No `—enable-dp-attention`; `pp_size == 1`; disables overlap scheduler & mixed chunked prefill
STANDALONE	Smaller draft LLM (token-level)	Yes	`—speculative-algorithm STANDALONE` + `—speculative-draft-model-path …`	Does not support `—enable-dp-attention`
NGRAM	Ngram cache from previous tokens	No	`—speculative-algorithm NGRAM`	CUDA-only; no `—enable-dp-attention`; disables overlap scheduler & mixed chunked prefill

Performance Highlights

Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding. For further details please see the EAGLE3 paper.

Method	Throughput (tokens/s)
SGLang (w/o speculative, 1x H100)	158.34 tokens/s
SGLang + EAGLE-2 (1x H100)	244.10 tokens/s
SGLang + EAGLE-3 (1x H100)	373.25 tokens/s

EAGLE Decoding

To enable EAGLE speculative decoding the following parameters are relevant:

Parameter	Description	Default
`—speculative-draft-model-path`	Draft model path/weights. Typically required for EAGLE/EAGLE3 and STANDALONE. For some MTP-enabled models, this can be omitted.	`None`
`—speculative-num-steps`	Depth of autoregressive drafting. Increases speculation range but risks rejection cascades.	Auto (`5` for Llama/Grok; `3` for many other models)
`—speculative-eagle-topk`	Branching factor per step. Improves candidate diversity and acceptance rate, but increases memory/compute consumption.	Auto (`4` for Llama/Grok; `1` for many other models)
`—speculative-num-draft-tokens`	Maximum parallel verification capacity. Allows deeper tree evaluation but increases GPU memory usage.	Auto (`8` for Llama/Grok; `4` for many other models). If `topk=1`, it is adjusted to `num_steps + 1`.
`—speculative-accept-threshold-single`	Acceptance threshold for single-token verification. Lower values accept more aggressively.	`1.0`
`—speculative-accept-threshold-acc`	Accumulated acceptance threshold across steps.	`1.0`
`—speculative-attention-mode`	Attention mode for speculative operations (`prefill` or `decode`), affecting both target verification and draft extension.	`”prefill”`
`—speculative-draft-attention-backend`	Override attention backend for the draft model.	`None` (same as target)
`—speculative-dsa-topk-backend`	Select the DSA indexer top-k backend for speculative draft workers independently of `—dsa-topk-backend`. Options are `sgl-kernel`, `torch`, and `flashinfer`; `torch` requires `SGLANG_DSA_FUSE_TOPK=false`.	`sgl-kernel`
`—speculative-draft-model-quantization`	Quantization method for the draft model. Use `“unquant”` to force no quantization even when the target model is quantized.	Same as target model
`—speculative-draft-model-revision`	Specific revision/commit of the draft model to load.	`None` (auto-set to `“main”` when `—speculative-draft-model-path` is set and revision is omitted)
`—speculative-draft-load-format`	Load format for the draft model weights.	`None`

These parameters are mostly the same for EAGLE-2 and EAGLE-3. --speculative-token-map is ignored for EAGLE-3 models. For --speculative-num-steps, --speculative-eagle-topk, and --speculative-num-draft-tokens: leave all three unset to use auto-tuning, or set all three explicitly when tuning. If you use EAGLE with --speculative-eagle-topk 1 and your acceptance rate varies across requests, see Adaptive Speculative Decoding. You can find the best combinations of these parameters with bench_speculative.py.

EAGLE-2 Decoding

You can enable EAGLE-2 Decoding by setting --speculative-algorithm EAGLE and choosing an appropriate model. Launch the server:

Command

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-2 Decoding with `torch.compile`

You can optionally enable torch.compile to apply kernel-level optimizations (operator fusion, autotune) to the draft model. The actual speedup depends on your hardware, model architecture, and batch size. In some configurations (e.g., small draft models on H100 where cuBLAS is already optimal and CUDA graphs are enabled), the benefit may be negligible. We recommend benchmarking with and without this flag on your specific setup to verify whether it helps. To enable it, add --enable-torch-compile and optionally set --torch-compile-max-bs:

Command

python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --enable-torch-compile \
    --torch-compile-max-bs 8 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

By employing a truncated high-frequency token vocabulary in the draft model, EAGLE speculative decoding reduces lm_head computational overhead while accelerating the pipeline without quality degradation. For more details, check out the paper. In our implementation, set --speculative-token-map to enable the optimization. You can get the high-frequency tokens in FR-Spec from this model. Or you can obtain high-frequency tokens by directly downloading these tokens from this repo. Thanks for the contribution from Weilin Zhao and Zhousx.

Command

python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --dtype float16 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

EAGLE-3 Decoding

You can enable EAGLE-3 decoding by setting --speculative-algorithm EAGLE3 and choosing an appropriate model.

Command

python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --dtype float16 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Multi Token Prediction

We support MTP (Multi-Token Prediction) in SGLang by using speculative decoding. We use XiaomiMiMo/MiMo-7B-RL as an example here (for DeepSeek MTP usage, refer to DeepSeek-V3.2 cookbook §4.2.3).

Command

python3 -m sglang.launch_server \
    --model XiaomiMiMo/MiMo-7B-RL \
    --host 0.0.0.0 \
    --trust-remote-code \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 1 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 2 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

For DSA-based MTP, draft workers default to the sgl-kernel top-k backend. Use --speculative-dsa-topk-backend to override the draft independently of --dsa-topk-backend for the target model. Send a request:

Example

import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "XiaomiMiMo/MiMo-7B-RL",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
}

response = requests.post(url, json=data)
print(response.json())

UNO decoding

UNO reuses the target transformer for both passes of each speculative decode cycle instead of loading a separate draft model. During the draft forward, the first row of each request uses the base weights and the remaining B - 1 rows use a trained UNO LoRA. Target verification is entirely base-only. UNO provides two sampling modes:

Linear constructs and verifies one chain of draft tokens, similar to DFlash’s proposal layout.
Tree expands multiple candidates at each draft depth and uses SGLang’s EAGLE tree-verification path.

For UNO modes, the three numbers are B/K/V: draft-forward width, candidates kept per expansion, and verification width. Thus, Linear 8/1/8 uses an 8-token draft width, one candidate per expansion, and an 8-token verification width. UNO’s K controls proposal-tree breadth and is unrelated to the request sampling parameter top_k. The command-line mapping depends on the mode:

In linear mode, B and V are both --speculative-num-draft-tokens; set --speculative-num-steps 1 and --speculative-eagle-topk 1.
In tree mode, B is --speculative-num-steps + 1, K is --speculative-eagle-topk, and V is --speculative-num-draft-tokens.

Use an adapter trained for the exact target-model checkpoint. The current integration has been validated with Qwen/Qwen3-8B; unsupported LoRA target layers are rejected at startup. Set the adapter’s local path or Hugging Face repository before starting the server:

Linear 8/1/8

Command

export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B"

sglang serve \
    --model-path Qwen/Qwen3-8B \
    --speculative-algorithm UNO \
    --uno-lora-path "$UNO_LORA_PATH" \
    --speculative-num-steps 1 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 8 \
    --attention-backend fa3 \
    --tp 1 \
    --host 0.0.0.0 \
    --port 30000

Tree 8/32/8

The following command selects tree mode because K is greater than one:

Command

export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B"

sglang serve \
    --model-path Qwen/Qwen3-8B \
    --speculative-algorithm UNO \
    --uno-lora-path "$UNO_LORA_PATH" \
    --speculative-num-steps 7 \
    --speculative-eagle-topk 32 \
    --speculative-num-draft-tokens 8 \
    --attention-backend fa3 \
    --tp 1 \
    --host 0.0.0.0 \
    --port 30000

Send requests through the standard OpenAI-compatible API:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Explain speculative decoding."}],
    max_tokens=128,
)

print(response.choices[0].message.content)

Key requirements and limitations

UNO requires CUDA with FA3 for prefill and decode, tensor and pipeline parallel sizes of 1, and no DP attention or context parallelism.
--uno-lora-path loads UNO’s fixed internal adapter and cannot be combined with request-selectable Multi-LoRA serving.
UNO manages its own stochastic verification; do not set --speculative-use-rejection-sampling.
Grammar decoding, returned logprobs or hidden states, sampling penalties, min_p, logit bias, custom logit processors, strict thinking, and deterministic inference are not yet supported.
In tree mode, both speculative acceptance thresholds must remain at 1.0, V must be at least B and at most 128, and V * K must not exceed 2048. SGLang validates the remaining tree-capacity and EAGLE parent-representation constraints at startup.
Mixed chunked prefill is disabled for UNO.
Ordinary overlap scheduling is supported. Tree mode does not yet support PDMux or the separate --enable-two-batch-overlap feature.

DFlash Decoding

SGLang also supports DFLASH speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as meta-llama/Llama-3.1-8B-Instruct with z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat. Relevant parameters:

Parameter	Description	Default
`—speculative-draft-model-path`	Required DFlash draft model path/weights.	`None`
`—speculative-num-draft-tokens`	DFlash verify block size.	Inferred from draft config, otherwise `16`
`—speculative-dflash-block-size`	Alias of `—speculative-num-draft-tokens` for DFlash.	`None`
`—speculative-dflash-draft-window-size`	Draft KV sliding-window size. Must be `>= speculative-num-draft-tokens` when set.	`None`

Command

python3 -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Write a quicksort implementation in Python."},
    ],
    temperature=0,
    max_tokens=128,
)

print(response.choices[0].message.content)

Standalone Speculative Decoding (Small Draft Model)

Besides EAGLE/MTP, SGLang also supports token-level speculative decoding using a smaller draft model. Enable it with --speculative-algorithm STANDALONE and provide a draft model via --speculative-draft-model-path. Relevant parameters:

Parameter	Description	Default
`—speculative-draft-model-path`	Draft model weights (smaller than the target model).	`None`
`—speculative-num-steps`	Draft depth (how many steps the draft model runs autoregressively).	`3` (auto default for STANDALONE)
`—speculative-eagle-topk`	Branching factor (token candidates per step).	`1` (auto default for STANDALONE)
`—speculative-num-draft-tokens`	Verification capacity.	`4` (auto default for STANDALONE)
`—speculative-draft-model-quantization`	Quantization for the draft model. Use `“unquant”` to disable quantization on the draft even when the target is quantized.	Same as target

Note: Standalone speculative decoding currently does not support --enable-dp-attention.

Command

python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm STANDALONE \
    --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 2 \
    --speculative-num-draft-tokens 7 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Speculative Decoding V2 (Overlap Scheduler)

Speculative decoding runs the V2 speculative workers (e.g. StandaloneWorkerV2, EAGLEWorkerV2) with the overlap scheduler enabled by default. Pass --disable-overlap-schedule to fall back to the synchronous (non-overlap) path. Notes:

The overlap scheduler currently only supports --speculative-eagle-topk 1; set --speculative-eagle-topk 1 explicitly.
If you explicitly set --speculative-eagle-topk > 1, the server will error.
If you omit --speculative-eagle-topk, auto-tuning may pick topk > 1 for some models (e.g. Llama). This is incompatible with the overlap scheduler and may not always trigger an immediate config error, so set --speculative-eagle-topk 1 explicitly.

Command

python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm STANDALONE \
    --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Ngram Speculative Decoding

SGLang also supports ngram-based speculative decoding (no separate draft model). It retrieves draft tokens from an ngram cache built from previously generated tokens, and then verifies them with the target model. Enable it with:

--speculative-algorithm NGRAM

Ngram-specific parameters

Parameter	Description	Default
`—speculative-num-draft-tokens`	Number of draft tokens verified per step. If omitted, defaults to `min(—speculative-ngram-max-trie-depth, 12)`.	`12` (with default ngram settings)
`—speculative-ngram-min-bfs-breadth`	Minimum BFS breadth.	`1`
`—speculative-ngram-max-bfs-breadth`	Maximum BFS breadth.	`10`
`—speculative-ngram-match-type`	Ngram tree-building mode: `“BFS”` for recency-based expansion or `“PROB”` for frequency-based expansion.	`”BFS”`
`—speculative-ngram-max-trie-depth`	Maximum suffix length stored and matched by the ngram trie.	`18`
`—speculative-ngram-capacity`	Cache capacity (number of entries).	`10,000,000`

Notes:

Ngram speculative decoding only supports CUDA.
It currently does not support --enable-dp-attention.
It disables the overlap scheduler and mixed chunked prefill.
If --speculative-ngram-max-bfs-breadth > 1 (thus speculative_eagle_topk > 1) and page_size > 1, use --attention-backend flashinfer; otherwise the server will error.
Optional: set SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True to force greedy verification.

Command

python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm NGRAM \
    --speculative-num-draft-tokens 16 \
    --speculative-ngram-max-bfs-breadth 10 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning

Send a request:

Example

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)

Full Parameter Reference

Below is a comprehensive list of all speculative decoding parameters available in SGLang:

Core parameters

Parameter	Type	Default	Description
`—speculative-algorithm`	`str`	`None`	Algorithm to use: `UNO`, `DFLASH`, `EAGLE`, `EAGLE3`, `STANDALONE`, `NGRAM`, `NEXTN` (alias of `EAGLE`)
`—speculative-draft-model-path`	`str`	`None`	Path to the draft model weights
`—uno-lora-path`	`str`	`None`	UNO-only path or Hugging Face repository for the draft LoRA checkpoint
`—speculative-draft-model-revision`	`str`	`None`	Specific revision/commit of the draft model (`“main”` is auto-used when draft path is set and revision is omitted)
`—speculative-draft-load-format`	`str`	`None`	Load format for draft model weights
`—speculative-num-steps`	`int`	`None` (auto-chosen when omitted)	Autoregressive drafting depth. Some algorithms auto-tune it; tree UNO requires it and uses `B = speculative_num_steps + 1`.
`—speculative-eagle-topk`	`int`	`None` (auto-chosen when omitted)	Branching factor per drafting step. Some algorithms auto-tune it; UNO resolves an omitted value to `1`, and values greater than one select UNO tree mode.
`—speculative-num-draft-tokens`	`int`	`None` (auto-chosen when omitted)	Maximum number of draft tokens for verification. Some algorithms auto-tune it; UNO requires it and uses it as linear width or tree verification width `V`.
`—speculative-dflash-block-size`	`int`	`None`	DFlash-only alias of `—speculative-num-draft-tokens`
`—speculative-dflash-draft-window-size`	`int`	`None`	DFlash-only draft KV sliding-window size
`—speculative-accept-threshold-single`	`float`	`1.0`	Single-token acceptance threshold
`—speculative-accept-threshold-acc`	`float`	`1.0`	Accumulated acceptance threshold
`—speculative-token-map`	`str`	`None`	Path to FR-Spec high-frequency token map
`—speculative-attention-mode`	`str`	`”prefill”`	Attention mode for speculative operations (`“prefill”` or `“decode”`)
`—speculative-draft-attention-backend`	`str`	`None`	Override attention backend for the draft model
`—speculative-dsa-topk-backend`	`str`	`sgl-kernel`	DSA indexer top-k backend for speculative draft workers, independent of `—dsa-topk-backend` (`sgl-kernel`, `torch`, or `flashinfer`)
`—speculative-moe-runner-backend`	`str`	`None`	MoE runner backend for the draft model
`—speculative-moe-a2a-backend`	`str`	`None`	MoE all-to-all backend for the draft model
`—speculative-draft-model-quantization`	`str`	Same as target	Quantization for the draft model (`“unquant”` to disable)

Ngram-specific parameters

Parameter	Type	Default	Description
`—speculative-ngram-min-bfs-breadth`	`int`	`1`	Minimum BFS breadth
`—speculative-ngram-max-bfs-breadth`	`int`	`10`	Maximum BFS breadth
`—speculative-ngram-match-type`	`str`	`”BFS”`	Ngram tree-building mode: `“BFS”` for recency-based expansion or `“PROB”` for frequency-based expansion
`—speculative-ngram-max-trie-depth`	`int`	`18`	Maximum suffix length stored and matched by the ngram trie
`—speculative-ngram-capacity`	`int`	`10,000,000`	Cache capacity

Environment variables

Variable	Default	Description
`SGLANG_NGRAM_FORCE_GREEDY_VERIFY`	`False`	Force greedy verification for ngram decoding

Parameter	Description
`—enable-multi-layer-eagle`	Enable multi-layer EAGLE (auto-enabled for MiMoV2 and Step3p5 models)
`—enable-torch-compile`	Enable `torch.compile` for kernel-level optimizations
`—torch-compile-max-bs`	Maximum batch size for `torch.compile`

OOM Troubleshooting

[!WARNING] Out of Memory (OOM)? Speculative decoding may increase GPU memory usage because the draft tree, CUDA graphs, and verification-related buffers consume additional VRAM. If you encounter OOM errors, try the following adjustments.

Step 1: Lower static memory fraction (most effective)

Command

--mem-fraction-static 0.5   # when omitted, this value is auto-computed

--mem-fraction-static controls the memory budget for model weights + KV cache pool.
Lowering it directly increases dynamic headroom for activations and CUDA graph buffers.
If omitted, SGLang auto-estimates this value from other settings, and those auto settings can still be too aggressive for some workloads.

Step 2: Reduce CUDA graph batch size

Command

# Fewer CUDA graph captures = less memory reserved
--cuda-graph-max-bs-decode 4   # or even 2 for tight memory situations

If omitted, --cuda-graph-max-bs-decode is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.

Step 3: Reduce draft tree size

These three parameters directly control how much memory the draft tree consumes:

Command

# Before (aggressive, high memory)
--speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64

# After (conservative, lower memory)
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Step 4: Limit concurrent requests

Command

# Fewer concurrent requests lowers in-flight load and can reduce OOM risk
--max-running-requests 4

Quick OOM recovery recipe

If you’re hitting OOM and just want something that works, start with this minimal configuration and scale up:

Command

python3 -m sglang.launch_server \
    --model <your-model> \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path <your-draft-model> \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --cuda-graph-max-bs-decode 2 \
    --mem-fraction-static 0.5 \
    --max-running-requests 4 \
    --log-level warning

Then gradually increase --speculative-num-draft-tokens, --speculative-eagle-topk, and --cuda-graph-max-bs-decode. Increase --mem-fraction-static last, only after the run is stable.

References

EAGLE process is as follows:

Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_{k+1})$ .
The next token is then sampled from $p_{k+2}=\text{LMHead}(f_{k+1})$ . Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the speculative_eagle_topk parameter—to ensure a more coherent connection of context, and are given as input again.
In SGLang’s EAGLE-2 implementation, the draft tree is expanded for the configured steps and then reranked to select the top speculative_num_draft_tokens final nodes as draft tokens.
EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.

This enhances drafting accuracy by operating on features instead of tokens for more regular inputs and by additionally passing tokens from the next timestep to reduce sampling randomness. For more details, see the EAGLE-2 and EAGLE-3 papers. For guidance on how to train your own EAGLE model please see the EAGLE repo. For EAGLE-3 training specifically, check out SpecForge, the SGLang team’s training framework designed for EAGLE-3 speculative decoding models with seamless porting to SGLang serving. See the SpecForge documentation and blog post for details.

Basic Usage

Advanced Features

Supported Models

Developer Guide

References

Summary

Jump to sections

Quick guidance

Method comparison (mini table)

Performance Highlights

EAGLE Decoding

EAGLE-2 Decoding

EAGLE-2 Decoding with `torch.compile`

EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

EAGLE-3 Decoding

Multi Token Prediction

UNO decoding

Linear 8/1/8

Tree 8/32/8

Key requirements and limitations

DFlash Decoding

Standalone Speculative Decoding (Small Draft Model)

Speculative Decoding V2 (Overlap Scheduler)

Ngram Speculative Decoding

Ngram-specific parameters

Full Parameter Reference

Core parameters

Ngram-specific parameters

Environment variables

OOM Troubleshooting

Step 1: Lower static memory fraction (most effective)

Step 2: Reduce CUDA graph batch size

Step 3: Reduce draft tree size

Step 4: Limit concurrent requests

Quick OOM recovery recipe

References

​Summary

​Jump to sections

​Quick guidance

​Method comparison (mini table)

​Performance Highlights

​EAGLE Decoding

​EAGLE-2 Decoding

​EAGLE-2 Decoding with torch.compile

​EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

​EAGLE-3 Decoding

​Multi Token Prediction

​UNO decoding

​Linear 8/1/8

​Tree 8/32/8

​Key requirements and limitations

​DFlash Decoding

​Standalone Speculative Decoding (Small Draft Model)

​Speculative Decoding V2 (Overlap Scheduler)

​Ngram Speculative Decoding

​Ngram-specific parameters

​Full Parameter Reference

​Core parameters

​Ngram-specific parameters

​Environment variables

​Other related flags

​OOM Troubleshooting

​Step 1: Lower static memory fraction (most effective)

​Step 2: Reduce CUDA graph batch size

​Step 3: Reduce draft tree size

​Step 4: Limit concurrent requests

​Quick OOM recovery recipe

​References

Summary

Jump to sections

Quick guidance

Method comparison (mini table)

Performance Highlights

EAGLE Decoding

EAGLE-2 Decoding

EAGLE-2 Decoding with `torch.compile`

EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

EAGLE-3 Decoding

Multi Token Prediction

UNO decoding

Linear 8/1/8

Tree 8/32/8

Key requirements and limitations

DFlash Decoding

Standalone Speculative Decoding (Small Draft Model)

Speculative Decoding V2 (Overlap Scheduler)

Ngram Speculative Decoding

Ngram-specific parameters

Full Parameter Reference

Core parameters

Ngram-specific parameters

Environment variables

Other related flags

OOM Troubleshooting

Step 1: Lower static memory fraction (most effective)

Step 2: Reduce CUDA graph batch size

Step 3: Reduce draft tree size

Step 4: Limit concurrent requests

Quick OOM recovery recipe

References