Fused Kernels - SGLang Documentation

Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under sglang/kernels/ops/diffusion. This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is --quality, described below.

Numerical contracts and quality tiers

Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so “close enough” and “bit-exact” are different products here. The quality switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions: Bit-exact — mounted unconditionally. The kernel reproduces every rounding boundary of the eager chain, so torch.equal holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch’s vectorized_layer_norm_kernel down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer’s CuTe-DSL RMSNormKernel fragment order and shfl.bfly fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch. Not bit-exact — request-gated. These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for quality="extra-high" and quality="high" requests, at batch boundaries, all-or-nothing per transformer. The default quality="lossless" runs the unmodified reference chain. Model/checkpoint-native. Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but quality does not select or undo those choices. Selection-equivalent routing — enabled unconditionally. LingBot Video’s fused group-limited top-k returns the same selected expert-id set as its guarded CUDA torch.topk(..., sorted=False) reference chain. The order of those ids is not part of either path’s contract. Because the selected experts are unchanged, this path does not depend on the request quality tier.

A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path.

The quality levels are cumulative:

`quality`	Included optimization set
`lossless`	The selected deployment’s reference path plus every unconditional bit-exact replacement
`extra-high`	Everything in `lossless`, plus request-gated DiT and VAE kernel fusions; this level does not itself enable sparse, caching, or another approximate path
`high`	Everything in `extra-high`, plus any model-owned high-only optimization, such as an audited Cache-DiT policy or lower-precision VAE decode

If a model has no eligible request-gated fusion, extra-high can execute the same path as lossless. Likewise, high adds only the model-specific high-only paths that the active pipeline implements.

quality is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier.

Enabling the request-gated set

sglang generate --model-path MODEL_PATH --prompt "..." --quality extra-high

The server default stays lossless; the OpenAI-compatible endpoints carry it per request. Images:

curl -X POST http://${HOST}:${PORT}/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'

Video, same field:

curl -X POST http://${HOST}:${PORT}/v1/videos \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}'

The quality field in a video response body is unrelated. It is Sora-compatible response metadata and is always reported as "standard"; it does not reflect the sampling quality the request ran with.

quality participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused.

Do not combine request-gated DiT fusions with --enable-breakable-cuda-graph. BCG warmup captures the lossless module branches before an extra-high or high request mounts its DiT fusions, so replay would bypass the requested kernels. SGLang rejects this combination for models with eligible DiT quality sites. Models whose request-gated path changes only VAE decode remain allowed because BCG captures the DiT only.

These fusion families mount under both quality="extra-high" and quality="high":

Fusion	What it folds
Linear + tanh-GELU	Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip
Wan NVFP4 linear + GELU	Bias-add and GELU fused into Wan’s NVFP4 FFN projection output
Qwen-Image added-QKV	Added Q/K/V projections fused into joint-buffer production
LayerNorm + modulate	`layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate
LTX-2 RMSNorm + modulate	`rms_norm(x) * (1 + scale) + shift` in one launch
Gate RMSNorm (BF16-native)	`RMSNorm + tanh + mul + add` in one pass
HunyuanVideo strided QK RMSNorm	Per-head QK RMSNorm over the packed QKV layout
LingBot Video fused RMSNorm	Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels
LingBot Video per-token gated residual + RMSNorm modulate	Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels
SANA-Video BF16-input linear attention	Keeps the first linear-attention GEMM’s inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32
FLUX-family VAE fast paths	Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines
Wan VAE RMSNorm + SiLU	Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d`

Kernel inventory

45 operators are registered in the kernel registry across 51 implementations (some operators carry several backends). Backends are named by provenance, not device: KDA identifies Kernel Design Agents implementations, JIT compiles under nvcc and hipcc, TRITON identifies Triton sources, CUTE_DSL needs CUTLASS, FLYDSL is ROCm gfx950 only, and AOT comes from the sgl_kernel wheel. Per-operator capability metadata determines which devices can load each implementation.

Normalization

Operator	Backend	Contract	Replaces
`rmsnorm_scale_shift`	Triton	bit-exact	RMSNorm + `* (1 + scale) + shift` (4 kernels)
`scale_residual_norm_scale_shift`	KDA / Triton / CuTe-DSL / FlyDSL	bit-exact (Triton) or backend-specific close contract	the above plus the preceding `residual + gate * update`
`scale_residual_norm_scale_shift_nvfp4`	JIT CUDA	matches the selected NVFP4 producer contract	Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization
`layernorm_modulate`	Triton	bit-exact	affine-free LayerNorm + adaLN modulate
`try_fused_fp32_layernorm_bf16`	KDA / Triton	bit-exact against the verified native dispatch	LingBot World causal FP32 LayerNorm on Hopper with FP32 affine or adaLN post-ops, including BF16 input/output conversions; other devices retain native LayerNorm
`qk_head_layernorm`	Triton	bit-exact	per-head LayerNorm on q/k
`qk_rmsnorm_native`	Triton	bit-exact	Z-Image per-head QK RMSNorm
`norm_scale_shift`	KDA / CuTe-DSL / FlyDSL	backend-specific close contract	LN-or-RMS + scale/shift, many broadcast modes
`rmsnorm_scale`, `rmsnorm_tanh_residual`	Triton	bf16-native statistics	`RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)`
`apply_group_norm_silu`	Triton	close	`GroupNorm + SiLU`, NCHW-contiguous
`group_norm_silu_4d`, `group_norm_silu_rows`	Triton	close	channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels_last end to end with no `nchwToNhwc` transposes
`wan_rmsnorm_silu`	Triton	close	Wan VAE `channels_last_3d` RMSNorm + SiLU
`wan_norm_silu_post`	JIT CUDA	bit-exact, first-sight verified	Wan VAE FP32 divide, scale, affine, and SiLU after the unchanged native L2 reduction; dense NCDHW and `channels_last_3d`

adaLN modulation and gating

Operator	Backend	Contract	Replaces
`modulate_scale_shift`	JIT CUDA	bit-exact	`x * (1 + scale) + shift`
`residual_gate_add`	KDA (JIT CUDA)	bit-exact	`residual + gate * update`
`try_fused_scaled_residual_bf16`	Triton	bit-exact against the verified native dispatch	LingBot World causal BF16 residual with an FP32 per-frame gate; skips the unused normalized output before camera conditioning
`timestep_embedding`	JIT CUDA	close	sinusoidal timestep embedding
`temb_table_slices`	Triton	bit-exact	see note below
`ltx2_ada_values`	Triton	bit-exact	LTX-2 nine-way adaLN value split, slices come out contiguous

LingBot World camera conditioning also uses modulate_scale_shift with one affine row per token. The causal normalization and residual paths verify each new signature outside CUDA graph capture and fall back to their native chains on mismatch or unsupported layouts. These paths preserve lossless arithmetic; kernel replay support does not enable BCG for a model configuration that disables it.

temb_table_slices is worth knowing about. The eager (scale_shift_table + temb.float()).chunk(6, dim=2) materializes roughly 8 GB of fp32 at 704p/121f and hands six strided slices downstream, whose .contiguous() calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops.

RoPE and QK-norm

Operator	Backend	Contract	Replaces
`fused_inplace_qknorm_rope`	JIT CUDA	one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True`	separate QK-norm kernel + RoPE
`flux2_qkv_epilogue`	KDA (JIT CUDA)	bit-exact against its selected BF16 reference chain	FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing
`flux2_strided_qknorm_rope`	Triton	bit-exact against native tiled RMSNorm followed by FlashInfer interleaved RoPE; verified per signature	FLUX.2 Klein single-stream packed Q/K copies + QK RMSNorm + RoPE on Hopper, BF16, 24 heads of dimension 128
`qwen_qkv_epilogue`	JIT CUDA	bit-exact against its selected BF16 reference chain	Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+
`rope_rotate_half`	Triton	bit-exact	`chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection
`interleaved_rope_fp64`	JIT CUDA	bit-exact	paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels
`helios_qk_rope`	JIT CUDA	bit-exact	paired in-place Helios Q/K RoPE with transposed frequency layout
`ltx2_qknorm_split_rope`	KDA (JIT CUDA)	close (validated on B200)	LTX-2 QK-norm + split RoPE
`ltx25_decoder_rope`	JIT CUDA	bit-exact	paired LTX-2.5 decoder 3D RoPE from cached compact axis tables
`hunyuan_qkv_rope_pack`	Triton	bit-exact	QKV pack and RoPE in one pass

The Klein strided path preserves the BF16 rounding between RMSNorm and RoPE, including signed zeros. It reads packed Q/K views without changing the adjacent V or MLP values. Unsupported layouts, devices, or RMSNorm dispatches use the original helper. A new signature is verified outside CUDA Graph capture; an unverified capture falls back. SGLANG_ENABLE_FUSED_QKNORM_ROPE=0 also disables this path. This kernel does not enable model-level BCG support for FLUX.2.

Activation

Operator	Backend	Contract	Replaces
`silu_mul`	Triton	bit-exact	`F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat`
`bias_silu`, `bias_glu`	Triton	bit-exact	Sana GLUMB conv bias + SiLU / GLU post-processing
`linear_gelu_tanh`	AOT (cublasLt)	not bit-exact, request-gated	bias-add and tanh-GELU folded into the GEMM epilogue

Attention

Operator	Backend	Notes
`sparse_linear_attn_fwd`	Triton	block-map, compression, and forward for sparse linear attention
`bigdn`	Triton	Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B’s MMA folds the identity-add in

MoE routing

Operator	Backend	Contract	Replaces
`group_limited_topk`	Triton	selected expert-id set matches the guarded CUDA reference; output order is unspecified	LingBot Video’s per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k

Data movement

Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces.

Operator	Backend	Replaces
`usp_merge_heads`	JIT CUDA	USP all-to-all output head merge (`permute` + `contiguous`)
`pack_qkv_destination_major`	Triton	Ulysses destination-major QKV pack
`varlen_pack_qkv`, `varlen_scatter_to_padded`	Triton	varlen gather/scatter around the masked attention path
`varlen_pack_segmented_qkv`	Triton	varlen gather from a virtual prefix/main Q/K/V sequence
`joint_qkv_cat`	Triton	Joy Image Edit’s three image/text Q/K/V concatenations, including strided packed-V inputs
`causal_conv3d_cat_pad`	KDA (JIT CUDA) / Triton	causal Conv3d `cat` + `pad`
`cat_pad_channels_last_3d`	Triton	Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass
`dup_up3d_add`	Triton	`repeat_interleave + permute().contiguous() + add`

Joy Image Edit uses the joint-copy path for eligible CUDA FP16/BF16 image tensors of at least 32 MiB. It preserves image-first token order, copies values without arithmetic, and verifies each new shape/stride signature against the native concatenations before enabling it. Small inputs, unsupported layouts, gradient-bearing inputs, and unverified signatures during graph capture use the native path. This does not enable model-level BCG support. For large Hopper BF16 image Q/K with 32 heads of width 128, Joy also uses the existing out-of-place QK-Norm + RoPE kernel to read the packed projection directly. This removes two input copies while retaining the original CUDA arithmetic and contiguous outputs. Each new signature is checked bitwise; inputs remain intact if the operation fails. Other shapes and platforms, compilation, unverified capture, and SGLANG_ENABLE_FUSED_QKNORM_ROPE=0 retain the original helper.

Quantized layout producers

These kernels preserve the quantized checkpoint path’s selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint.

Operator	Backend	Replaces
`flux2_layernorm_modulate_fp8_quant`	KDA (Triton)	FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output
`flux2_token_cat_fp8`	KDA (Triton)	FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization
`flux2_token_cat_nvfp4`	JIT CUDA	FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization

Coverage by model

Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal.

Model	Fused paths
FLUX.1	LN+modulate, modulate, residual-gate add, linear+GELU
FLUX.2	LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers
Qwen-Image	linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer
GLM-Image	LN+modulate, per-head qk LN, residual-gate add, linear+GELU
ERNIE-Image	RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add
Joy Image Edit	Bitwise image/text QKV concatenation; strided image QK-Norm + RoPE on Hopper
Z-Image	BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm
Ideogram 4	gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add
LTX-2	QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU
LTX-2.5 decoder	paired 3D RoPE with shared axis-table cache
HunyuanVideo / Helios	QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE
LingBot Video MoE	Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=extra-high` or `quality=high`
Sana	LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add
SANA-Video	Packed QKV/KV; paired fp64 interleaved RoPE; GLUMB bias+SiLU / bias+GLU in eager and BCG; LN+modulate and residual-gate add during BCG; BF16-input linear attention at `quality=extra-high` or `quality=high`
Sana-WM	bidirectional gated delta-net, fused QK inverse-RMS
Wan / LongLive 2 VAE	temb table slices (Wan); VAE cat+pad and DupUp3D add; lossless FP32 normalization post-ops; full `channels_last_3d` RMSNorm+SiLU at `quality=extra-high` or `quality=high`
Cosmos3	QK-norm + RoPE + KV packing; rounded T2I fusion also supports Hopper TP2/SP1 Super-Text2Image with the 5120-wide SwiGLU architecture
Krea2 / MiniMax-H3	QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation)
FLUX.2 VAE / HunyuanVAE / latent upsampler	GroupNorm + SiLU (channels-last two-pass for FLUX.2)

Inspecting what is registered

Every kernel is described by a KernelSpec in the process-wide registry, so the inventory is queryable without importing any backend:

from sglang.kernels.registry import registry

diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")]
for spec in registry.get("diffusion.scale_residual_norm_scale_shift"):
    print(spec.backend, spec.target, spec.capabilities)

Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several:

from sglang.kernels import select_kernel, KernelBackend

fn = select_kernel(
    "diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL
).load()

Importing the kernels

Runtime code imports from the package, never from a submodule:

from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact

Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call can_use_<op>(...) first and fall back to the reference chain when it returns False; the kernel raises on an unsupported input rather than silently returning None. The package README.md carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed.

References

Performance Optimization
Attention Backends
Quantization
Profiling
sglang/kernels/ops/diffusion — source and selection matrix
RFC #29630 — the unified sglang.kernels namespace

​Numerical contracts and quality tiers

​Enabling the request-gated set

​Kernel inventory

​Normalization

​adaLN modulation and gating

​RoPE and QK-norm

​Activation

​Attention

​MoE routing

​Data movement

​Quantized layout producers

​Coverage by model

​Inspecting what is registered

​Importing the kernels

​References

Numerical contracts and quality tiers

Enabling the request-gated set

Kernel inventory

Normalization

adaLN modulation and gating

RoPE and QK-norm

Activation

Attention

MoE routing

Data movement

Quantized layout producers

Coverage by model

Inspecting what is registered

Importing the kernels

References