Performance Optimization - SGLang Documentation

Use this page as the starting point for SGLang Diffusion performance work. It separates performance levers into two decision classes:

Output-preserving / lossless-style: system settings that should preserve model behavior while changing residency, parallelism, kernels, or scheduling.
Quality-tradeoff / lossy or approximate: techniques that can change the denoising path, numerical representation, or generated output.

The docs use “output-preserving” instead of promising bit-exact “lossless” because different kernels, GPU types, or precision paths can still introduce small numerical differences. The decision boundary is whether the optimization intentionally trades quality or output equivalence for speed.

Start Here

Pick a serving or generation mode from Deployment and Performance Modes. --performance-mode auto is the default; use speed when the model fits in GPU memory and latency matters most, memory when GPU memory is the bottleneck, and manual when every performance flag should be explicit.
Choose the right attention backend from Attention Backends.
Use Sequence Parallelism only when the model and video shape benefit from sequence splitting.
Use Inference Batching for concurrent compatible requests during serving.
Use Profiling before changing several levers at once.

Choose a request quality tier

--quality is cumulative: a broader tier never drops an optimization from a stricter tier.

Tier	Optimization boundary
`lossless` (default)	The selected deployment’s reference execution plus all unconditional bit-exact replacements
`extra-high`	Everything in `lossless`, plus only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths
`high`	Everything in `extra-high`, plus model-owned high-only paths such as an audited Cache-DiT policy or lower-precision VAE decode

Use extra-high when you want to isolate fusion wins from approximate acceleration. A tier may be a no-op when the active model has no eligible path. Separately configured quantization, attention, or caching options still apply. See Fused Kernels for the current request-gated families and their numerical contracts.

Output-Preserving / Lossless-Style Levers

These settings should preserve model behavior while changing residency, parallelism, kernels, or scheduling. They are the first choices for production tuning.

Lever	Use when	Docs
`—performance-mode`	You want a safe preset for speed or memory without overriding explicit flags.	Deployment and Performance Modes
Breakable CUDA graph	A supported pipeline serves a fixed set of shapes and eager execution is launch-bound.	CLI reference
Offload, FSDP, CFG parallelism	GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck.	Deployment and Performance Modes
Sequence parallelism	Long image/video sequences need sequence-level parallelism.	Sequence Parallelism
`—encoder-parallel`	Text/image encoding is a visible share of the request and the DiT replica sits idle during it.	Encoder Parallelism
Attention backend	Kernel choice dominates DiT latency or memory.	Attention Backends
Fused kernels	You want to know which elementwise chains are already fused, or to opt into the request-gated set.	Fused Kernels
Dynamic batching	Serving many compatible requests concurrently.	Inference Batching
`—async-output-save`	Image/video encoding and disk writes leave the GPU idle between queued requests.	Asynchronous output saving

Asynchronous output saving

Enable --async-output-save to overlap output saving with the next request’s GPU work during serving. Frame materialization, image/video encoding, and disk writes run on a background thread. The flag is disabled by default and does not change the model’s generation settings or the requested output format. Use it when you have concurrent requests and saving takes a noticeable share of each request, such as PNG images or large videos from a fast model. A client that waits for each response before sending the next request has no work to overlap.

Enable

The feature requires NVIDIA CUDA and the native SGLang backend in monolithic serving mode. After following Installation, you can start this example on one GPU with the public Hugging Face model Tongyi-MAI/Z-Image-Turbo:

Command

sglang serve \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --backend sglang \
  --num-gpus 1 \
  --batching-max-size 1 \
  --async-output-save

Send concurrent requests using the OpenAI-compatible API. --batching-max-size 1 keeps each forward separate while allowing requests to queue. Grouped and dynamically batched forwards currently keep synchronous output saving.

Scope and tradeoffs

Only the output rank performs asynchronous saving, and only for requests that save files and return their paths (save_output=true and return_file_paths_only=true in the sampling parameters).
Warmup requests, raw-frame responses, requests with save_output=false, and requests with perf_dump_path set or SGLANG_DIFFUSION_STAGE_LOGGING=1 keep the synchronous path. This includes the frame-return path used by /rollout/generate.
Each reply is sent from the saving thread as soon as its output is saved, even while the next request’s forward is running. Measure both throughput and response latency against the flag disabled.
Up to two pending finalizations retain request outputs in GPU memory. The scheduler waits for capacity before starting another forward; leave memory headroom for the pending outputs as well as inference.
Per-request peak memory metrics can include allocations from the next request. Asynchronous finalization skips the usual per-request allocator cache release.

Compare the flag on and off with the same model, output shape, format, and client concurrency. Keep it disabled when the reply latency or extra memory outweighs the throughput gain.

Quality-Tradeoff / Lossy Or Approximate Levers

These techniques can change the denoising path, numerical representation, or generated output. They are useful after you have a baseline and an acceptance criterion for quality.

Lever	Tradeoff	Docs
Cache-DiT	Skips selected DiT block or step computation based on cache decisions.	Cache-DiT
TeaCache	Reuses residuals when consecutive denoising steps are similar enough.	TeaCache
Progressive resolution	Runs early denoising at lower latent resolution for supported pipelines.	Progressive Resolution Generation
Quantization	Uses lower-precision transformer weights or activations.	Quantization

Practical Order

Establish a baseline with the target model, resolution, frame count, step count, and GPU type.
Select --performance-mode and explicit residency or parallelism flags.
Compare quality=lossless with quality=extra-high to isolate the request-gated fusion set.
Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to --warmup-resolutions and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches.
Tune attention backend and batching for the deployment pattern.
Profile if the bottleneck is unclear.
Add quality=high, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target.

Per-model tuning starting points

Warmup and breakable CUDA graph (BCG) solve different problems. --warmup-mode request runs a warmup copy derived from the first request to prime one-time compilation and caches; it does not remove recurring Python launches from each denoising step. BCG captures supported DiT segments and can reduce that recurring launch overhead for captured shapes. Use the table below as a first experiment, then keep a lever only when profiling confirms that it addresses the active bottleneck.

Observed bottleneck or constraint	Example models	First experiment	What to verify
The first execution of a shape pays substantial compilation or cache setup	ERNIE-Image-Turbo, GLM-Image, FastWan / TurboWan family, FLUX.2-klein-4B, LingBot-World, LongLive, Cosmos3, SANA, LongCat-Image-Edit-Turbo	Compare `--warmup-mode request` with `off`	Separate warmup time, first-real-request latency, and warmed steady-state latency. Request-based warmup is primarily a benchmark aid and still consumes time before the first real request completes.
A supported fixed-shape pipeline shows recurring host-launch gaps between GPU kernels	GLM-Image, Z-Image-Turbo, Qwen-Image-2512, Ideogram-4, LongCat-Image, LTX-2 / LTX-2.3, SANA-1.5 1.6B, JoyEcho	Compare eager execution with `--enable-breakable-cuda-graph`	Confirm graph capture and replay in the server log, then compare steady-state latency and GPU memory. Pass additional production shapes through `--warmup-resolutions`; without it, BCG captures only the model’s default warmup resolution.
The model does not fit while both DiT stages are resident	Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, LingBot-Video-MoE	Start with `--dit-layerwise-offload`	Verify peak GPU memory first, then measure the transfer overhead. Add warmup separately if first-run compilation is also material.
Dynamic masks, metadata, or image conditions make graph replay ineffective	Qwen-Image-Edit family, FireRed-Image-Edit	Keep eager execution as the baseline	Profile before attempting BCG. Dynamic per-step host work can outweigh graph replay savings, and unsupported pipelines fall back to eager execution.
Kernels or collectives already dominate a many-step workload	Qwen-Image, FLUX.1-dev, FLUX.2-dev, LTX-2, Wan2.1 14B, HunyuanVideo, MOVA	Warm the baseline, then profile before enabling another lever	A small host gap limits the benefit available from BCG. Prioritize kernels, attention, parallelism, or residency according to the trace.
A one-shot or very few-step workload cannot amortize warmup	FastVideo-FastH3 (4-step)	Start with `--warmup-mode off`	Compare end-to-end latency including warmup. Use request warmup only when intentionally separating cold-start setup from the measured request.

The example assignments come from single-GPU NVIDIA B300/GB300 profiles and are directional rather than an exhaustive compatibility list. A model can match more than one row as resolution, frame count, step count, parallelism, or GPU type changes; the measured bottleneck takes precedence over the model name. BCG is enabled only for model and pipeline configurations accepted by the runtime support check. It captures the default warmup shape automatically. Use --warmup-resolutions for additional served resolutions; for video and variable prompt lengths, set --warmup-num-frames and --bcg-text-buckets to cover the intended workload. Requests that do not match a captured signature fall back to eager execution. Warmup and BCG do not intentionally trade output quality for speed, but that is not a guarantee of byte-identical output. Different execution paths can introduce numerical differences, and some pipelines customize the scheduler or schedule used by a synthetic warmup request. Before production rollout, compare output hashes when bitwise stability is required, otherwise run the project’s quality acceptance check. Performance results are configuration-dependent. Record the exact checkpoint revision, GPU, precision, resolution, frame count, step count, parallelism, command line, and whether the measurement includes warmup. Re-profile with Profiling on the target workload rather than transferring a percentage from another model or shape.

Diagnostics

Profiling is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers.

​Start Here

​Choose a request quality tier

​Output-Preserving / Lossless-Style Levers

​Asynchronous output saving

​Enable

​Scope and tradeoffs

​Quality-Tradeoff / Lossy Or Approximate Levers

​Practical Order

​Per-model tuning starting points

​Diagnostics

​References