> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Attention Backends > Select and configure attention backends for SGLang diffusion pipelines. This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them. ## Overview Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders). When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). * **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel. * **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA. * **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA. * **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA. * **MPS**: always uses PyTorch SDPA. * **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA. ## Backend options For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
CLI value Enum value Notes
`fa` / `fa3` / `fa4` `FA` FlashAttention. fa3/fa4 are normalized to fa during argument parsing (ServerArgs.**post\_init**).
`torch_sdpa` `TORCH_SDPA` PyTorch scaled\_dot\_product\_attention.
`sliding_tile_attn` `SLIDING_TILE_ATTN` Sliding Tile Attention (STA). Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` `SAGE_ATTN` Requires sageattention. On Hopper (SM90), PyPI sageattention==2.2.0 is unsupported because it lacks the upstream SM90 binding fix. Install pip install --force-reinstall git+[https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4](https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4) --no-build-isolation. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream setup.py.
`sage_attn_3` `SAGE_ATTN_3` Requires SageAttention3 installed per upstream instructions.
`sol_attn` `SOL_ATTN` Requires the upstream sol-attn package. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). BF16, head dim 128. Configure via --attention-backend-config.
`video_sparse_attn` `VIDEO_SPARSE_ATTN` Requires vsa. Configure sparsity via --attention-backend-config.
`vmoba_attn` `VMOBA_ATTN` Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
`aiter` `AITER` Requires aiter.
aiter\_sage AITER\_SAGE Requires aiter.
sla\_attn SLA\_ATTN Sparse Linear Attention. Requires SpargeAttn. Install with pip install git+[https://github.com/thu-ml/SpargeAttn.git](https://github.com/thu-ml/SpargeAttn.git) --no-build-isolation.
sage\_sla\_attn SAGE\_SLA\_ATTN SageAttention + Sparse Linear Attention. Requires SpargeAttn (same install as SLA).
`sparse_video_gen_2_attn` `SPARSE_VIDEO_GEN_2_ATTN` Requires svg. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).
laser\_attn LASER\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
block\_sparse\_attn BLOCK\_SPARSE\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
rain\_fusion\_attn RAIN\_FUSION\_ATTN Requires attentions which can be installed with sgl\_kernel\_npu; available only for NPU.
## Selection priority The selection order in `runtime/layers/attention/selector.py` is: 1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)` 2. Component override from `--component-attention-backends` while that component is being constructed 3. CLI `--attention-backend` (`ServerArgs.attention_backend`) 4. Auto selection (platform capability, dtype, and installed packages) ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: * A path to a JSON or YAML configuration file. * A JSON string (e.g., `'{"sparsity": 0.5}'`). * Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`). ### Supported Configuration Parameters **Sliding Tile Attention (`sliding_tile_attn`)**
Parameter Type Description Default
`mask_strategy_file_path` `str` **Required.** Path to the mask strategy JSON file. -
`sta_mode` `str` Mode of STA. `STA_inference`
`skip_time_steps` `int` Number of steps to use full attention before switching to sparse attention. `15`
**Video Sparse Attention (`video_sparse_attn`)**
Parameter Type Description Default
`sparsity` `float` Validation sparsity (0.0 - 1.0). `0.0`
**V-MoBA (`vmoba_attn`)**
Parameter Type Description Default
`temporal_chunk_size` `int` Chunk size for temporal dimension. -
`temporal_topk` `int` Top-K tokens to select in temporal dimension. -
`spatial_chunk_size` `list[int]` Chunk size for spatial dimension (H, W). -
`spatial_topk` `int` Top-K tokens to select in spatial dimension. -
`st_chunk_size` `list[int]` Chunk size for spatiotemporal dimension (T, H, W). -
`st_topk` `int` Top-K tokens to select in spatiotemporal dimension. -
`moba_select_mode` `str` Selection mode (e.g., `threshold`). `threshold`
`moba_threshold` `float` Threshold value for selection. `0.25`
`moba_threshold_type` `str` Type of thresholding (e.g., `query_head`). `query_head`
`first_full_step` `int` Number of initial steps to use full attention. `12`
`first_full_layer` `int` Number of initial layers to use full attention. `0`
`temporal_layer` `int` Number of temporal layers. `1`
`spatial_layer` `int` Number of spatial layers. `1`
`st_layer` `int` Number of spatiotemporal layers. `1`
**Block Sparse Attention (`block_sparse_attn`)**
Parameter Type Description Default
`skip_first_steps` `int` Number of steps to use laser attention before switching to sparse attention. `10`
`sparsity` `float` The sparsity coefficient must be in the range (0, 1). `0.2`
**Sol-Attn (`sol_attn`)**
Parameter Type Description Default
`tau` `float` Routing threshold scale. Higher values select fewer exact KV blocks. `1.0`
`thresh_type` `str` Threshold mode: `diag` or `exact`. `diag`
`sink_tokens` `int` Exact KV sink length for prefix tokens such as text/audio rows. `0`
`sink_start` `int` Start index of the exact KV sink range. `0`
`dense_steps` `int` Use dense attention for the first N denoising steps. `10`
`dense_layers` `str` Layer indices kept dense, e.g. `0,1` or `0-2`. `0,1`
`dense_backend` `str` Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate. `fa`
`kv_splits` `int | str` KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences. `auto`
## Platform support matrix
Backend CUDA ROCm XPU MUSA MPS NPU Notes
`fa` Yes Yes ✅ ✅ ❌ ✅ CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to torch\_sdpa. No extra installations are required for NPU
`torch_sdpa` Yes Yes Yes Yes ✅ ✅ Most compatible option across platforms.
`sliding_tile_attn` Yes No No No ❌ ❌ CUDA-only. Requires st\_attn. Configure via --attention-backend-config.
`sage_attn` Yes No No Yes ❌ ❌ Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3` Yes No No No ❌ ❌ CUDA-only (optional dependency).
`sol_attn` Yes No No No ❌ ❌ CUDA-only. Requires sol-attn. Install with pip install git+[https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse\_backends](https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends). Configure via --attention-backend-config.
`video_sparse_attn` Yes No No No ❌ ❌ CUDA-only. Requires vsa. Configure sparsity via --attention-backend-config.
sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
sage\_sla\_attn Yes No No No ❌ ❌ CUDA-only. Requires SpargeAttn.
vmoba\_attn Yes No No No ❌ ❌ CUDA-only. Requires kernel.attn.vmoba\_attn.vmoba. Configure via --attention-backend-config.
aiter ❌ ✅ No No ❌ ❌ Requires aiter.
aiter\_sage ❌ ✅ No No ❌ ❌ Requires aiter.
`sparse_video_gen_2_attn` Yes No No No ❌ ❌ CUDA-only. Requires svg.
laser\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Uses SDPA if seqlen less than 2048.
block\_sparse\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu. Configuration via --attention-backend-config.
rain\_fusion\_attn ❌ ❌ ❌ ❌ ❌ ✅ NPU-only. Requires attentions from sgl\_kernel\_npu Configuration via --attention-backend-config.
## Usage ### Select a backend via CLI ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa ``` ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend torch_sdpa ``` ### Override one component Use component overrides when a specific module needs different attention semantics from the main transformer: ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. ### Per-request override (denoise loop) A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the `attention_backend_override` sampling param. Valid values are the exact/drop-in dense kernels — `fa`, `torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the dynamic-batch signature, so requests with different backends never share a batch. ```bash theme={null} sglang generate \ --model-path \ --prompt "..." \ --attention-backend-override sage_attn ``` ```python theme={null} client.images.generate( model="", prompt="...", extra_body={"attention_backend_override": "sage_attn"}, ) ``` Incompatible server settings **reject the request** (with a server log) instead of silently falling back: breakable CUDA graphs and `torch.compile` bake the attention kernel into a captured/traced graph; sparse server backends (`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set `dense_backend=sage_attn` to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on `torch_sdpa`: ```bash theme={null} sglang generate \ --model-path MiniMaxAI/MiniMax-H3 \ --model-variant fl2va \ --attention-backend sol_attn \ --attention-backend-config dense_backend=sage_attn,dense_steps=10 \ --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \ --prompt "A cat walking on a sunny beach, gentle waves." \ --save-output ``` Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`. ### Using Sliding Tile Attention (STA) ```bash theme={null} # Pass the mask strategy file path via config sglang generate \ --model-path \ --prompt "..." \ --attention-backend sliding_tile_attn \ --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json" ``` ### Notes for ROCm / MPS * ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. * MPS: the platform implementation always uses `torch_sdpa`.