Cache-DiT Acceleration - SGLang Documentation

SGLang integrates Cache-DiT, a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 1.69x inference speedup with minimal quality loss.

Overview

Cache-DiT uses intelligent caching strategies to skip redundant computation in the denoising loop:

DBCache (Dual Block Cache): Dynamically decides when to cache transformer blocks based on residual differences
TaylorSeer: Uses Taylor expansion for calibration to optimize caching decisions
DMD Calibrator: An exponential-basis forecasting calibrator (Dynamic Mode Decomposition, not Distribution Matching Distillation) that serves as a drop-in alternative to TaylorSeer’s polynomial basis; strongest on flow-matching models
SCM (Step Computation Masking): Step-level caching control for additional speedup

Cache-DiT also ships SVDQuant W4A4 (int4 / NVFP4) dynamic quantization, which can be combined with DBCache caching (see Quantization).

Basic Usage

Cache-DiT is a per-request switch: each request decides whether to run cached or lossless, and requests with different Cache-DiT settings never share a batch. The SGLANG_CACHE_DIT_* environment variables remain available as server-wide defaults for requests that leave the switch unset. Enable it for a single generation:

sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A beautiful sunset over the mountains" \
    --enable-cache-dit true

Or per request against a running server, via the OpenAI-compatible API:

client.images.generate(
    model="Qwen/Qwen-Image",
    prompt="A beautiful sunset over the mountains",
    extra_body={
        "enable_cache_dit": True,
        # optional knob overrides for this request only
        "cache_dit_params": {"residual_diff_threshold": 0.12, "scm_preset": "fast"},
    },
)

enable_cache_dit accepts three states: true (on for this request), false (off for this request, overriding the server default), and unset (follow the SGLANG_CACHE_DIT_ENABLED server default). cache_dit_params accepts the DBCache knobs (Fn_compute_blocks, Bn_compute_blocks, max_warmup_steps, residual_diff_threshold, max_continuous_cached_steps, enable_taylorseer, taylorseer_order), the DMD knobs (enable_dmd, dmd_history, dmd_rank, dmd_ridge, dmd_svd_precision; DMD and TaylorSeer are mutually exclusive calibrators and cannot be enabled together), the SCM knobs (scm_preset, scm_compute_bins, scm_cache_bins, scm_policy), and a nested secondary dict with the DBCache knobs for the second transformer of dual-DiT models (unset secondary keys inherit the request’s primary values, then the SGLANG_CACHE_DIT_SECONDARY_* defaults). To make Cache-DiT the default for every request instead, export the environment variable when launching:

SGLANG_CACHE_DIT_ENABLED=true \
sglang serve --model-path Qwen/Qwen-Image

Diffusers Backend

Cache-DiT supports loading acceleration configs from a custom YAML file. For diffusers pipelines (diffusers backend), pass the YAML/JSON path via --cache-dit-config. This flow requires cache-dit >= 1.2.0 (cache_dit.load_configs).

Single GPU inference

Define a cache.yaml file that contains:

DBCache + TaylorSeer

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1

Then apply the config with:

sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config cache.yaml \
  --prompt "A beautiful sunset over the mountains"

DBCache + TaylorSeer + SCM (Step Computation Mask)

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
  # Must set the num_inference_steps for SCM. The SCM will automatically
  # generate the steps computation mask based on the num_inference_steps.
  # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking
  num_inference_steps: 28
  steps_computation_mask: fast

DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
  num_inference_steps: 28
  steps_computation_mask: fast
  enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.

DBCache + DMD Calibrator

Instead of TaylorSeer, you can use the DMD calibrator: an exponential-basis forecasting calibrator that serves as a drop-in alternative to TaylorSeer’s polynomial basis. DMD models the cached feature stream as a linear dynamical system (Y_{t+1} ~= A @ Y_t), forecasts cached features from the fitted eigen-modes, and stays accurate over longer cache skips where polynomial extrapolation diverges. DMD here refers to Dynamic Mode Decomposition (Schmid 2010), not Distribution Matching Distillation. DMD works best on flow-matching models (e.g., FLUX), while TaylorSeer is often better on DDPM-style models — try both. DMD and TaylorSeer are mutually exclusive — enable only one of enable_dmd / enable_taylorseer:

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0 # Bn=0 since the DMD calibrator replaces the Bn calibrator
  residual_diff_threshold: 0.12
  enable_dmd: true
  dmd_history: 6 # snapshot window length, 5-6 typical
  dmd_svd_precision: "medium" # "low", "medium" or "high"

A dmd_history window of 5–6 snapshots is typically the sweet spot — longer histories do not always help, because the feature dynamics drift across timesteps. With fewer than 4 uniformly spaced snapshots available, DMD transparently falls back to the Taylor expansion it maintains internally. See the Cache-DiT DMD documentation for the mathematical principle and quantitative comparisons. A ready-made config is available at examples/configs/cache_dmd.yaml in the Cache-DiT repository. Apply it with the same --cache-dit-config flag:

sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config cache_dmd.yaml \
  --prompt "A beautiful sunset over the mountains"

Distributed inference

1D Parallelism

Define a parallelism only config yaml parallel.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  attention_backend: native

Then, apply the distributed inference acceleration config from yaml. ulysses_size: auto means that cache-dit will auto detect the world_size as the ulysses_size. Otherwise, you should manually set it as specific int number, e.g, 4. Then apply the distributed config with: (Note: please add --num-gpus N to specify the number of gpus for distributed inference)

sglang generate \
  --backend diffusers \
  --num-gpus 4 \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config parallel.yaml \
  --prompt "A futuristic cityscape at sunset"

2D Parallelism

You can also define a 2D parallelism config yaml parallel_2d.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  tp_size: 2
  attention_backend: native

Then, apply the 2D parallelism config from yaml. Here tp_size: 2 means using tensor parallelism with size 2. The ulysses_size: auto means that cache-dit will auto detect the world_size // tp_size as the ulysses_size.

3D Parallelism

You can also define a 3D parallelism config yaml parallel_3d.yaml file that contains:

Config

parallelism_config:
  ulysses_size: 2
  ring_size: 2
  tp_size: 2
  attention_backend: native

Then, apply the 3D parallelism config from yaml. Here ulysses_size: 2, ring_size: 2, tp_size: 2 means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.

Ulysses Anything Attention

To enable Ulysses Anything Attention, you can define a parallelism config yaml parallel_uaa.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_anything: true

Ulysses FP8 Communication

For device that don’t have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml parallel_fp8.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_float8: true

Async Ulysses CP

You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml parallel_async.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image.

Then, apply the config from yaml. Here ulysses_async: true means enabling async ulysses CP.

TE-P and VAE-P

You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml parallel_extra.yaml file that contains:

Config

parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]

Hybrid Cache and Parallelism

Define a hybrid cache and parallel acceleration config yaml hybrid.yaml file that contains:

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]

Then, apply the hybrid cache and parallel acceleration config from yaml.

sglang generate \
  --backend diffusers \
  --num-gpus 4 \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config hybrid.yaml \
  --prompt "A beautiful sunset over the mountains"

Attention Backend

In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file attention.yaml that only contains:

Config

attention_backend: "flash" # '_flash_3' for Hopper

Quantization

You can also specify the quantization config in the yaml file, required torchao>=0.16.0. For example, define a yaml file quantize.yaml that contains:

Config

quantize_config: # quantization configuration for transformer modules
  # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc.
  quant_type: "float8"
  # layers to exclude from quantization (transformer). layers that contains any of the
  # keywords in the exclude_layers list will be excluded from quantization. This is useful
  # for some sensitive layers that are not robust to quantization, e.g., embedding layers.
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false # whether to print verbose logs during quantization

Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example:

Command

sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --warmup-mode request \
  --cache-dit-config quantize.yaml \
  --enable-torch-compile \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --prompt "A beautiful sunset over the mountains"

SVDQuant (W4A4 int4 / NVFP4)

SVDQuant is Cache-DiT’s built-in W4A4 PTQ quantization (weights and activations in int4 or NVFP4, with smoothed low-rank branches). It can be freely combined with DBCache caching and the DMD calibrator for the largest speedups. ::::note SVDQuant requires a cache-dit build with CUDA extension support — a plain pip install cache-dit does NOT include it. Install one of:

Command

# Option 1: prebuilt CUDA 13 wheel
pip install cache-dit-cu13==<version> --no-deps

# Option 2: build from source with SVDQuant enabled
git clone https://github.com/vipshop/cache-dit
cd cache-dit
export CUDA_HOME=/usr/local/cuda
CACHE_DIT_BUILD_SVDQUANT=1 pip install ".[quantization]" --no-build-isolation

:::: Valid quant_type values are svdq_int4_r{32,64,128,256}_dq (int4 W4A4) and svdq_nvfp4_r{32,64,128,256}_dq (NVFP4 W4A4; requires a Blackwell GPU). Example config combining SVDQuant NVFP4 with DBCache + DMD (see examples/configs/blackwell/cache_dmd_svdq.yaml):

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_dmd: true
  dmd_history: 6
  dmd_svd_precision: "medium"
quantize_config:
  quant_type: "svdq_nvfp4_r128_dq" # nvfp4 for Blackwell; use svdq_int4_r128_dq for int4
  svdq_kwargs:
    quantize_device: "cuda"
    fused_mlp: true
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false

For int4 W4A4 (pre-Blackwell GPUs), the same config with quant_type: "svdq_int4_r128_dq" is available at examples/configs/cache_dmd_svdq.yaml (add runtime_kernel: "v2" to svdq_kwargs). Enable torch.compile for the best SVDQuant performance, and make sure --warmup-steps covers the compile warmup (use the same value as --num-inference-steps):

Command

sglang generate \
  --backend diffusers \
  --model-path black-forest-labs/FLUX.1-dev \
  --num-inference-steps=28 \
  --warmup-mode request \
  --warmup-steps 28 \
  --cache-dit-config cache_dmd_svdq.yaml \
  --enable-torch-compile \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --prompt "A beautiful sunset over the mountains"

You can verify from the log that the quantization is active: [Cache-DiT] SVDQuant Type: svdq_nvfp4_r128_dq, Rank: 128.

Combined Configs: Cache + Parallelism + Quantization

You can also combine all the above configs together in a single yaml file combined.yaml that contains:

Config

cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]
quantize_config:
  quant_type: "float8"
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false

Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization.

Advanced Configuration

DBCache Parameters

DBCache controls block-level caching behavior:

Parameter	Env Variable	Default	Description
Fn	`SGLANG_CACHE_DIT_FN`	1	Number of first blocks to always compute
Bn	`SGLANG_CACHE_DIT_BN`	0	Number of last blocks to always compute
W	`SGLANG_CACHE_DIT_WARMUP`	4	Warmup steps before caching starts
R	`SGLANG_CACHE_DIT_RDT`	0.24	Residual difference threshold
MC	`SGLANG_CACHE_DIT_MC`	3	Maximum continuous cached steps

TaylorSeer Configuration

TaylorSeer improves caching accuracy using Taylor expansion:

Parameter	Env Variable	Default	Description
Enable	`SGLANG_CACHE_DIT_TAYLORSEER`	false	Enable TaylorSeer calibrator
Order	`SGLANG_CACHE_DIT_TS_ORDER`	1	Taylor expansion order (1 or 2)

DMD Calibrator Configuration

DMD (Dynamic Mode Decomposition, Schmid 2010 — not Distribution Matching Distillation) is an exponential-basis forecasting calibrator and a drop-in alternative to TaylorSeer’s polynomial basis. At each full-compute step it records a snapshot of the computed features; at a cached step it identifies a linear propagator from the recent snapshot window (one economy SVD with rank truncation, then eigendecomposition) and forecasts the current features via eigenvalue powers — cheap to advance, and stable over longer cache skips where polynomial extrapolation diverges. It typically improves both speed and quality over pure DBCache. DMD and TaylorSeer are mutually exclusive (enabling both raises a ValueError); DMD is best for flow-matching models, TaylorSeer for DDPM-style ones. See the Cache-DiT DMD documentation for details:

Parameter	Env Variable	Default	Description
Enable	`SGLANG_CACHE_DIT_DMD`	false	Enable the DMD calibrator
History	`SGLANG_CACHE_DIT_DMD_HISTORY`	6	Snapshot window length; 5-6 typical. Needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer
Rank	`SGLANG_CACHE_DIT_DMD_RANK`	0	SVD truncation rank; 0 = automatic (drop modes below 1e-4 of the leading singular value)
Ridge	`SGLANG_CACHE_DIT_DMD_RIDGE`	1e-8	Tikhonov regularization added to the inverted singular values
SVD Precision	`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`	medium	SVD precision: “low”, “medium” or “high”

Usage (SGLD backend, env-driven):

Command

SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_DMD=true \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A curious raccoon in a forest"

On the diffusers backend, enable DMD from the yaml config instead (enable_dmd: true in cache_config, see Diffusers Backend); DMD can also be set per request via cache_dit_params: {"enable_dmd": true}.

Combined Configuration Example

DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters simultaneously:

Command

SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A curious raccoon in a forest"

SCM (Step Computation Masking)

SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and which to use cached results. SCM Presets SCM is configured with presets:

Preset	Compute Ratio	Speed	Quality
`none`	100%	Baseline	Best
`slow`	~75%	~1.3x	High
`medium`	~50%	~2x	Good
`fast`	~35%	~3x	Acceptable
`ultra`	~25%	~4x	Lower

Usage

Command

SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_PRESET=medium \
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A futuristic cityscape at sunset"

Custom SCM Bins For fine-grained control over which steps to compute vs cache:

Command

SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A futuristic cityscape at sunset"

SCM Policy

Policy	Env Variable	Description
`dynamic`	`SGLANG_CACHE_DIT_SCM_POLICY=dynamic`	Adaptive caching based on content (default)
`static`	`SGLANG_CACHE_DIT_SCM_POLICY=static`	Fixed caching pattern

Environment Variables

All Cache-DiT parameters can also be configured via environment variables, which act as the server-wide defaults for requests that don’t set enable_cache_dit / cache_dit_params. See Environment Variables for the complete list.

Supported Models

SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:

Model Family	Example Models
Wan	Wan2.1, Wan2.2
Flux	FLUX.1-dev, FLUX.2-dev
Z-Image	Z-Image-Turbo
Qwen	Qwen-Image, Qwen-Image-Edit, Qwen-Image 2.1
Hunyuan	HunyuanVideo
MiniMax	MiniMax-H3 (T2VA, FL2VA, and Ref2VA)

Performance Tips

Start with defaults: The default parameters work well for most models
Use TaylorSeer: It typically improves both speed and quality
Tune R threshold: Lower values = better quality, higher values = faster
SCM for extra speed: Use medium preset for good speed/quality balance
Warmup matters: Higher warmup = more stable caching decisions

Limitations

SGLang-native pipelines: Distributed Cache-DiT paths exist for supported pipelines. Hybrid SP+TP configurations add communication and cache coordination overhead, so validate them on the target model and hardware before using them as production defaults.
DiT layerwise offload: Compatible. Skipped blocks are not streamed, and the first layer after a skip may sync-load. Still incompatible with --use-fsdp-inference.
SCM minimum steps: SCM requires >= 8 inference steps to be effective. Some pipelines report steps - 1 NFEs (for example MiniMax-H3 num_inference_steps=8 is 7 NFEs), which trips the upstream steps_mask assertion; use at least 9 requested steps or custom bins.
Model support: The model must be registered in Cache-DiT’s BlockAdapterRegister or have an SGLang custom block adapter.

Troubleshooting

SCM disabled for low step count

For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache acceleration still works.

SVDQuant unavailable or load failure

SVDQuant cases raise svdq_is_available() = False or undefined symbol: ... materialize_cow_storage ... when the installed cache-dit has no CUDA extension, or the prebuilt wheel was compiled against an incompatible torch. Fix: reinstall from the cache-dit-cu13 wheel matching your torch version, or build cache-dit from source with CACHE_DIT_BUILD_SVDQUANT=1 (see Quantization). Quick self-check:

Command

python -c "from cache_dit.quantization.svdquant import svdq_is_available, svdq_get_load_error as e; print(svdq_is_available(), e())"

​Overview

​Basic Usage

​Diffusers Backend

​Single GPU inference

​Distributed inference

​Hybrid Cache and Parallelism

​Attention Backend

​Quantization

​SVDQuant (W4A4 int4 / NVFP4)

​Combined Configs: Cache + Parallelism + Quantization

​Advanced Configuration

​DBCache Parameters

​TaylorSeer Configuration

​DMD Calibrator Configuration

​Combined Configuration Example

​SCM (Step Computation Masking)

​Environment Variables

​Supported Models

​Performance Tips

​Limitations

​Troubleshooting

​SCM disabled for low step count

​SVDQuant unavailable or load failure

​References

Overview

Basic Usage

Diffusers Backend

Single GPU inference

Distributed inference

Hybrid Cache and Parallelism

Attention Backend

Quantization

SVDQuant (W4A4 int4 / NVFP4)

Combined Configs: Cache + Parallelism + Quantization

Advanced Configuration

DBCache Parameters

TaylorSeer Configuration

DMD Calibrator Configuration

Combined Configuration Example

SCM (Step Computation Masking)

Environment Variables

Supported Models

Performance Tips

Limitations

Troubleshooting

SCM disabled for low step count

SVDQuant unavailable or load failure

References