Quantization

​

Quick Reference

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --component-weights-paths.transformer black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "a curious pikachu"

sglang generate \
  --model-path /path/to/base-model \
  --component-paths.transformer /path/to/quantized-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion"

​

Quantized Component Repositories

• load through an SGLang quantization implementation;
• delegate a standard component to Transformers or Diffusers;
• fail closed in a native plain-state loader when the format cannot be restored.

​

Quant Families

​

Causal KV-Cache Quantization

pip install "sglang[diffusion,diffusion-qvg]"
pip install --no-deps quant-videogen==0.1.0

sglang serve \
  --model-path robbyant/lingbot-world-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --kv-cache-quant int4 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false

​

Storage Policy

​

How PRQ Works

x = centroid_1 + residual_1
residual_1 = centroid_2 + residual_2
...
x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual)

​

Quality And Performance

​

Tuning Options

​

Online Quantization

​

FP8 Online Quantization

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --component-quantizations.transformer fp8 \
  --prompt "a beautiful sunset" \
  --save-output

​

MXFP4 Online Quantization

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --component-quantizations.transformer mxfp4 \
  --prompt "a beautiful sunset" \
  --save-output

​

Kitchen INT8

pip install comfy-kitchen

sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --component-quantizations.transformer kitchen_int8 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output

​

Skipping Layers

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization fp8 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization mxfp4 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output

​

Transformers-managed Quantized Components

sglang serve \
  --model-path black-forest-labs/FLUX.1-dev \
  --component-paths.text_encoder_2 \
    diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2

​

Validated ModelOpt Checkpoints

​

ModelOpt FP8

​

Usage Examples

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output

sglang generate \
  --model-path hunyuanvideo-community/HunyuanVideo \
  --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer \
  --height 544 --width 960 --num-frames 17 \
  --prompt "A cinematic shot of a red sports car driving through rain at night" \
  --save-output

sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output

sglang generate \
  --model-path Qwen/Qwen-Image-Edit-2511 \
  --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer \
  --image-path /path/to/input.png \
  --prompt "Turn the scene into a warm watercolor illustration" \
  --save-output

​

Notes

• --transformer-path is the primary-DiT convenience spelling for converted ModelOpt FP8 component repos or directories that already carry config.json.
• If the override repo or local directory contains its own config.json, SGLang reads the quantization config from that override instead of relying on the base model config.
• --transformer-weights-path still works when you intentionally point at raw weight files or a directory that should be metadata-probed as weights first.
• dit_layerwise_offload is supported for ModelOpt FP8 checkpoints.
• dit_cpu_offload still stays disabled for ModelOpt FP8 checkpoints.
• The layerwise offload path now preserves the non-contiguous FP8 weight stride expected by the runtime FP8 GEMM path.
• On disk, the quantization config stays quant_method=modelopt with quant_algo=FP8; the modelopt-fp8 label in this document is a support family name, not a serialized config key.
• To build the converted checkpoint yourself from a ModelOpt diffusers export, use python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer.

​

ModelOpt NVFP4

​

Usage Examples

sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output

sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output

SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --width 2048 --height 2048 \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output

​

Notes

• Use --transformer-path for mixed ModelOpt NVFP4 transformer repos or local directories that already include config.json.
• Use --transformer-weights-path for raw NVFP4 exports, individual safetensors files, or repo layouts that should be treated as weights first.
• For dual-transformer pipelines such as Wan2.2-T2V-A14B-Diffusers, the primary --transformer-path override targets only transformer. Use a per-component override such as --transformer-2-path only when you intentionally want a non-default transformer_2.
• On Blackwell, the diffusion ModelOpt NVFP4 path defaults to FlashInfer TensorRT-LLM FP4 GEMM (flashinfer_trtllm).
• The published Qwen Image NVFP4 exports keep the img_mod/txt_mod modulation projections and first/last transformer blocks in BF16.
• Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the validated ModelOpt exports were faster than BF16 at 2048x2048 with SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass, while 1024x1024 remained BF16-faster.
• Direct --model-path loading is the canonical path for full Qwen Image ModelOpt NVFP4 repos and a compatibility path for FLUX.2 NVFP4-style repos or local directories.
• If --transformer-weights-path is provided explicitly, it takes precedence over the compatibility --model-path flow.
• For local directories, SGLang first looks for *-mixed.safetensors, then falls back to loading from the directory.
• To force the diffusion ModelOpt FP4 path onto a different FlashInfer backend, set SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND. Supported values include flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm.
• On disk, the quantization config stays quant_method=modelopt with quant_algo=NVFP4; the modelopt-nvfp4 label here is again a documentation family name rather than a serialized config key.

​

GGUF

​

Usage

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --component-weights-paths.transformer \
    leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \
  --num-gpus 1 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --port 30010

​

Whether to offload the VAE

​

Constraints

​

Validated scope

​

Nunchaku (SVDQuant)

​

Install

pip install nunchaku

​

File Naming and Auto-Detection

• enable SVDQuant
• infer --quantization-precision
• infer --quantization-rank

​

Usage Examples

sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
  --prompt "a beautiful sunset" \
  --save-output

sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
  --enable-svdquant \
  --quantization-precision int4 \
  --quantization-rank 128 \
  --prompt "a beautiful sunset" \
  --save-output

​

Notes

• --transformer-weights-path is the primary-DiT convenience spelling used by Nunchaku examples. Older config names such as quantized_model_path are treated as compatibility aliases.
• Auto-detection only happens when the checkpoint basename matches svdq-(int4|fp4)_r{rank}.
• The CLI values are int4 and nvfp4. In filenames, the NVFP4 variant is written as fp4.
• Lightning checkpoints usually expect matching --num-inference-steps, such as 4 or 8.
• Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) or SM12x GPUs. Hopper (SM90) is currently rejected.

​

ModelSlim

• Installation
  # Clone repo and install msmodelslim:
  git clone https://gitcode.com/Ascend/msmodelslim.git
  cd msmodelslim
  bash install.sh
  
• Multimodal_sd quantization Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to Wan2.2-T2V-A14B to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card).

  Note: You can find pre-quantized validated models on modelscope/Eco-Tech.

  Run quantization using one-click quantization (recommended):
  msmodelslim quant \
    --model_path /path/to/wan2_2_float_weights \
    --save_path /path/to/wan2_2_quantized_weights \
    --device npu \
    --model_type Wan2_2 \
    --quant_type w8a8 \
    --trust_remote_code True
  
  For more detailed examples of quantization of models, as well as information about their support, see the examples section in ModelSLim repo.

  Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim.

• Auto-Detection and different formats For msmodelslim checkpoints, it’s enough to specify only --model-path, the detection of quantization occurs automatically for each layer using parsing of quant_model_description.json config. In the case of Wan2.2 only Diffusers weights storage format are supported, whereas modelslim saves the quantized model in the original Wan2.2 format. For conversion, use the one-step wan_repack.py script:
  python wan_repack.py \
    --model-type Wan2.2-TI2V-5B \
    --original-model-path {path_to_original_diffusers_model} \
    --quant-path {path_to_quantized_model} \
    --output-path {path_to_converted_model}
  
  Supported --model-type values: Wan2.2-TI2V-5B (single-transformer), Wan2.2-T2V-A14B and Wan2.2-I2V-A14B (Cascade dual-transformer). The script automatically handles: copying the base model, converting quantized weights to Diffusers format, and restoring config.json.
• Usage Example With auto-detected flow:
  sglang generate \
    --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \
    --prompt "a beautiful sunset" \
    --save-output
  
• Available Quantization Methods:
  • W4A4_DYNAMIC linear with online quantization of activations
  • W8A8 linear with offline quantization of activations
  • W8A8_DYNAMIC linear with online quantization of activations
  • W8A8_MXFP8 linear with offline quantization (msmodelslim pre-quantized weights)
  • mxfp8 linear with online quantization (--quantization mxfp8)
  • W4A4_MXFP4 / W4A4_MXFP4_DUALSCALE linear with offline quantization (msmodelslim pre-quantized weights)
  • mxfp4_npu linear with online quantization (--quantization mxfp4_npu)

​

MXFP8 Quantization

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a fox walking through neon rain" \
  --save-output

​

MXFP8 Offline Quantization (msmodelslim)

sglang generate \
  --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-mxfp8 \
  --prompt "a beautiful sunset" \
  --save-output

​

MXFP4 Online Quantization

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp4_npu \
  --prompt "a fox walking through neon rain" \
  --save-output

Hardware requirement: Ascend A5 series or newer. npu_dynamic_dual_level_mx_quant and npu_dual_level_quant_matmul are not available on A2/A3. Note: Online MXFP4 weight quantization is experimental. The offline msmodelslim flow uses pre-quantized weights and may produce different numerical results.

​

MXFP4 Offline Quantization (msmodelslim)

sglang generate \
  --model-path {path_to_converted_mxfp4_model} \
  --prompt "a beautiful sunset" \
  --save-output