Performance Optimization - SGLang Documentation

Use this page as the starting point for SGLang Diffusion performance work. It separates performance levers into two decision classes:

Output-preserving / lossless-style: system settings that should preserve model behavior while changing residency, parallelism, kernels, or scheduling.
Quality-tradeoff / lossy or approximate: techniques that can change the denoising path, numerical representation, or generated output.

The docs use “output-preserving” instead of promising bit-exact “lossless” because different kernels, GPU types, or precision paths can still introduce small numerical differences. The decision boundary is whether the optimization intentionally trades quality or output equivalence for speed.

Start Here

Pick a serving or generation mode from Deployment and Performance Modes. --performance-mode auto is the default; use speed when the model fits in GPU memory and latency matters most, memory when GPU memory is the bottleneck, and manual when every performance flag should be explicit.
Choose the right attention backend from Attention Backends.
Use Sequence Parallelism only when the model and video shape benefit from sequence splitting.
Use Inference Batching for concurrent compatible requests during serving.
Use Profiling before changing several levers at once.

Choose a request quality tier

--quality is cumulative: a broader tier never drops an optimization from a stricter tier.

Tier	Optimization boundary
`exact`	The selected deployment’s reference execution plus all unconditional bit-exact replacements. Output is bit-identical to the reference path in the same environment
`lossless` (default)	Everything in `exact`, plus request-gated DiT/VAE fusions that keep the reference math and the precision of every operand and accumulator, changing only where or in which order the rounding happens. Output matches the reference to rounding error, not bit for bit
`high`	Everything in `lossless`, plus model-owned paths that change what is computed — lower precision, sparsity, or reuse across steps, such as an audited Cache-DiT policy or lower-precision VAE decode

Bit-identical describes reproducibility, not quality: a lossless fast path is the same mathematical function, and its rounding is no further from the exact value than the reference’s own. On FLUX.1-dev the two tiers differ by 34.5 dB PSNR, while the same tier differs by 32.5 dB between one GPU and two — the tier moves the image less than the parallel layout does, which is why it is the default. Ask for exact when a run has to be byte-for-byte reproducible, or when bisecting a numerical bug. extra-high is accepted as the former name of lossless. Note that lossless itself was the former name of exact, and that it is now the default: a request that passed quality="lossless" for bit-identical output has to pass quality="exact", and a deployment that never set quality moves from the bit-exact path to this one. A tier may be a no-op when the active model has no eligible path. Separately configured quantization, attention, or caching options still apply. See Fused Kernels for the current request-gated families and their numerical contracts.

Library numerics below the tiers

PyTorch enables two approximations by default, underneath the whole tier system: torch.backends.cudnn.allow_tf32 truncates fp32 convolution inputs to 10 mantissa bits, and torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction lets a bf16 GEMM accumulate split-K partials below fp32. Both change what is computed, so by the definitions above they belong to high. They are process-global and a server mixes tiers across concurrent batches, so they cannot follow a request. The runtime sets both explicitly per worker and logs the result, instead of inheriting whatever the installed PyTorch defaults to:

Flag	Default	Effect of turning it off
`--allow-cudnn-tf32`	`true` (PyTorch’s own default)	fp32 convolutions run on fp32 CUDA cores instead of TF32 tensor cores. Affects the VAE decoders that default to `vae_precision="fp32"`
`--allow-bf16-reduced-precision-reduction`	`true` (PyTorch’s own default)	bf16 GEMMs accumulate split-K partials at fp32

The defaults keep PyTorch’s values so that exact is no slower than it was before these flags existed. A deployment that needs the strict contract — fp32 and bf16 math at reference precision throughout — passes both as false and accepts the slower convolutions.

Output-Preserving / Lossless-Style Levers

These settings should preserve model behavior while changing residency, parallelism, kernels, or scheduling. They are the first choices for production tuning.

Lever	Use when	Docs
`—performance-mode`	You want a safe preset for speed or memory without overriding explicit flags.	Deployment and Performance Modes
Breakable CUDA graph	A supported pipeline serves a fixed set of shapes and eager execution is launch-bound.	CLI reference
Offload, FSDP, CFG parallelism	GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck.	Deployment and Performance Modes
Sequence parallelism	Long image/video sequences need sequence-level parallelism.	Sequence Parallelism
`—encoder-parallel`	Text/image encoding is a visible share of the request and the DiT replica sits idle during it.	Encoder Parallelism
Attention backend	Kernel choice dominates DiT latency or memory.	Attention Backends
Fused kernels	You want to know which elementwise chains are already fused, or to opt into the request-gated set.	Fused Kernels
Dynamic batching	Serving many compatible requests concurrently.	Inference Batching
`—async-output-save`	Image/video encoding and disk writes leave the GPU idle between queued requests.	Asynchronous output saving

Asynchronous output saving

Enable --async-output-save to overlap output saving with the next request’s GPU work during serving. Frame materialization, image/video encoding, and disk writes run on a background thread. The flag is disabled by default and does not change the model’s generation settings or the requested output format. Use it when you have concurrent requests and saving takes a noticeable share of each request, such as PNG images or large videos from a fast model. A client that waits for each response before sending the next request has no work to overlap.

Enable

The feature requires NVIDIA CUDA and the native SGLang backend in monolithic serving mode. After following Installation, you can start this example on one GPU with the public Hugging Face model Tongyi-MAI/Z-Image-Turbo:

Command

sglang serve \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --backend sglang \
  --num-gpus 1 \
  --batching-max-size 1 \
  --async-output-save

Send concurrent requests using the OpenAI-compatible API. --batching-max-size 1 keeps each forward separate while allowing requests to queue. Grouped and dynamically batched forwards currently keep synchronous output saving.

Scope and tradeoffs

Only the output rank performs asynchronous saving, and only for requests that save files and return their paths (save_output=true and return_file_paths_only=true in the sampling parameters).
Warmup requests, raw-frame responses, requests with save_output=false, and requests with perf_dump_path set or SGLANG_DIFFUSION_STAGE_LOGGING=1 keep the synchronous path. This includes the frame-return path used by /rollout/generate.
Each reply is sent from the saving thread as soon as its output is saved, even while the next request’s forward is running. Measure both throughput and response latency against the flag disabled.
Up to two pending finalizations retain request outputs in GPU memory. The scheduler waits for capacity before starting another forward; leave memory headroom for the pending outputs as well as inference.
Per-request peak memory metrics can include allocations from the next request. Asynchronous finalization skips the usual per-request allocator cache release.

Compare the flag on and off with the same model, output shape, format, and client concurrency. Keep it disabled when the reply latency or extra memory outweighs the throughput gain.

Encoding during VAE decode

Native Wan, H3, and LTX video VAEs can hand completed frames to the shared MP4 encoder within the current request. Unlike --async-output-save, this does not require a second request to obtain a benefit. Eligible single-video, CUDA, file-path-only requests use this path automatically; raw frames, compilation, interpolation, upscaling, and custom post-decoding hooks keep the normal path. LTX’s diffusion decoder is not included. Ordinary VAEs without frame callbacks keep the existing GPU-direct saving path. Handing over a complete clip cannot overlap encoding with decode, and is not automatically enabled: it can add overhead to short requests. The shared DecodingStage.decode(..., on_frames=...) supports an explicit complete-clip hand-off for custom integrations. Custom decoding stages retain their original path unless they explicitly integrate the encoder, preserving output transforms and audio muxing. Image, audio, and mesh outputs are not MP4 inputs. LTX without temporal tiling hands over the complete clip after decode. This reduces CPU float-frame materialization but does not overlap VAE decoding with encoding. Existing temporal tiling can emit finalized, blended prefixes while later tiles decode. This optimization does not enable temporal tiling: changing the tile schedule can change the video. Keep the same decode, output compression, and x264 preset when measuring. Include time until the MP4 is fully written, not just the decoding-stage timer. The encoder bounds queued frames and frame-conversion intermediates, and a failed encode falls back to normal saving. Model authors can follow the frame hand-off contract.

Quality-Tradeoff / Lossy Or Approximate Levers

These techniques can change the denoising path, numerical representation, or generated output. They are useful after you have a baseline and an acceptance criterion for quality.

Lever	Tradeoff	Docs
Cache-DiT	Skips selected DiT block or step computation based on cache decisions.	Cache-DiT
TeaCache	Reuses residuals when consecutive denoising steps are similar enough.	TeaCache
Progressive resolution	Runs early denoising at lower latent resolution for supported pipelines.	Progressive Resolution Generation
Quantization	Uses lower-precision transformer weights or activations.	Quantization

Practical Order

Establish a baseline with the target model, resolution, frame count, step count, and GPU type.
Select --performance-mode and explicit residency or parallelism flags.
Compare quality=lossless with quality=lossless to isolate the request-gated fusion set.
Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to --warmup-resolutions and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches.
Tune attention backend and batching for the deployment pattern.
Profile if the bottleneck is unclear.
Add quality=high, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target.

Per-model tuning starting points

Warmup and breakable CUDA graph (BCG) solve different problems. --warmup-mode request runs a warmup copy derived from the first request to prime one-time compilation and caches; it does not remove recurring Python launches from each denoising step. BCG captures supported DiT segments and can reduce that recurring launch overhead for captured shapes. Use the table below as a first experiment, then keep a lever only when profiling confirms that it addresses the active bottleneck.

Observed bottleneck or constraint	Example models	First experiment	What to verify
The first execution of a shape pays substantial compilation or cache setup	ERNIE-Image-Turbo, GLM-Image, FastWan / TurboWan family, FLUX.2-klein-4B, LingBot-World, LongLive, Cosmos3, SANA, LongCat-Image-Edit-Turbo	Compare `--warmup-mode request` with `off`	Separate warmup time, first-real-request latency, and warmed steady-state latency. Request-based warmup is primarily a benchmark aid and still consumes time before the first real request completes.
A supported fixed-shape pipeline shows recurring host-launch gaps between GPU kernels	GLM-Image, Z-Image-Turbo, Qwen-Image-2512, Ideogram-4, LongCat-Image, LTX-2 / LTX-2.3, SANA-1.5 1.6B, JoyEcho	Compare eager execution with `--enable-breakable-cuda-graph`	Confirm graph capture and replay in the server log, then compare steady-state latency and GPU memory. Pass additional production shapes through `--warmup-resolutions`; without it, BCG captures only the model’s default warmup resolution.
The model does not fit while both DiT stages are resident	Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, LingBot-Video-MoE	Start with `--dit-layerwise-offload`	Verify peak GPU memory first, then measure the transfer overhead. Add warmup separately if first-run compilation is also material.
Dynamic masks, metadata, or image conditions make graph replay ineffective	Qwen-Image-Edit family, FireRed-Image-Edit	Keep eager execution as the baseline	Profile before attempting BCG. Dynamic per-step host work can outweigh graph replay savings, and unsupported pipelines fall back to eager execution.
Kernels or collectives already dominate a many-step workload	Qwen-Image, FLUX.1-dev, FLUX.2-dev, LTX-2, Wan2.1 14B, HunyuanVideo, MOVA	Warm the baseline, then profile before enabling another lever	A small host gap limits the benefit available from BCG. Prioritize kernels, attention, parallelism, or residency according to the trace.
A one-shot or very few-step workload cannot amortize warmup	FastVideo-FastH3 (8-step)	Start with `--warmup-mode off`	Compare end-to-end latency including warmup. Use request warmup only when intentionally separating cold-start setup from the measured request.

The example assignments come from single-GPU NVIDIA B300/GB300 profiles and are directional rather than an exhaustive compatibility list. A model can match more than one row as resolution, frame count, step count, parallelism, or GPU type changes; the measured bottleneck takes precedence over the model name. BCG is enabled only for model and pipeline configurations accepted by the runtime support check. It captures the default warmup shape automatically. Use --warmup-resolutions for additional served resolutions; for video and variable prompt lengths, set --warmup-num-frames and --bcg-text-buckets to cover the intended workload. Requests that do not match a captured signature fall back to eager execution. Warmup and BCG do not intentionally trade output quality for speed, but that is not a guarantee of byte-identical output. Different execution paths can introduce numerical differences, and some pipelines customize the scheduler or schedule used by a synthetic warmup request. Before production rollout, compare output hashes when bitwise stability is required, otherwise run the project’s quality acceptance check. Performance results are configuration-dependent. Record the exact checkpoint revision, GPU, precision, resolution, frame count, step count, parallelism, command line, and whether the measurement includes warmup. Re-profile with Profiling on the target workload rather than transferring a percentage from another model or shape.

Diagnostics

Profiling is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers.

​Start Here

​Choose a request quality tier

​Library numerics below the tiers

​Output-Preserving / Lossless-Style Levers

​Asynchronous output saving

​Enable

​Scope and tradeoffs

​Encoding during VAE decode

​Quality-Tradeoff / Lossy Or Approximate Levers

​Practical Order

​Per-model tuning starting points

​Diagnostics

​References