Quantization

​

Quick Reference

• --model-path: the base or original model
• --transformer-path: a quantized transformers-style transformer component directory that already contains its own config.json
• --transformer-weights-path: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "a curious pikachu"

sglang generate \
  --model-path /path/to/base-model \
  --transformer-path /path/to/quantized-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion"

​

Quant Families

​

Validated ModelOpt Checkpoints

​

ModelOpt FP8

​

Usage Examples

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-path BBuf/flux2-dev-modelopt-fp8-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path BBuf/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output

​

Notes

• --transformer-path is the canonical flag for converted ModelOpt FP8 transformer component repos or directories that already carry config.json.
• If the override repo or local directory contains its own config.json, SGLang reads the quantization config from that override instead of relying on the base model config.
• --transformer-weights-path still works when you intentionally point at raw weight files or a directory that should be metadata-probed as weights first.
• dit_layerwise_offload is supported for ModelOpt FP8 checkpoints.
• dit_cpu_offload still stays disabled for ModelOpt FP8 checkpoints.
• The layerwise offload path now preserves the non-contiguous FP8 weight stride expected by the runtime FP8 GEMM path.
• On disk, the quantization config stays quant_method=modelopt with quant_algo=FP8; the modelopt-fp8 label in this document is a support family name, not a serialized config key.
• To build the converted checkpoint yourself from a ModelOpt diffusers export, use python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer.

​

ModelOpt NVFP4

​

Usage Examples

sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --transformer-path BBuf/flux1-dev-modelopt-nvfp4-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

sglang generate \
  --model-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output

SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cudnn \
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path BBuf/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output

​

Notes

• Use --transformer-path for mixed ModelOpt NVFP4 transformer repos or local directories that already include config.json.
• Use --transformer-weights-path for raw NVFP4 exports, individual safetensors files, or repo layouts that should be treated as weights first.
• For dual-transformer pipelines such as Wan2.2-T2V-A14B-Diffusers, the primary --transformer-path override targets only transformer. Use a per-component override such as --transformer-2-path only when you intentionally want a non-default transformer_2.
• On Blackwell, the validated Wan2.2 ModelOpt NVFP4 path currently prefers FlashInfer FP4 GEMM via SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cudnn.
• This environment-variable override is a current workaround for NVFP4 cases where the default sglang JIT/CUTLASS sm100 path rejects a large-M shape at can_implement(). The intended long-term fix is to add a validated CUTLASS fallback for those shapes rather than rely on the override.
• Direct --model-path loading is a compatibility path for FLUX.2 NVFP4-style repos or local directories.
• If --transformer-weights-path is provided explicitly, it takes precedence over the compatibility --model-path flow.
• For local directories, SGLang first looks for *-mixed.safetensors, then falls back to loading from the directory.
• To force the generic diffusion ModelOpt FP4 path onto a specific FlashInfer backend, set SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND. Supported values include flashinfer_cudnn, flashinfer_cutlass, and flashinfer_trtllm.
• On disk, the quantization config stays quant_method=modelopt with quant_algo=NVFP4; the modelopt-nvfp4 label here is again a documentation family name rather than a serialized config key.

​

Nunchaku (SVDQuant)

​

Install

pip install nunchaku

​

File Naming and Auto-Detection

• enable SVDQuant
• infer --quantization-precision
• infer --quantization-rank

​

Usage Examples

sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
  --prompt "a beautiful sunset" \
  --save-output

sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
  --enable-svdquant \
  --quantization-precision int4 \
  --quantization-rank 128 \
  --prompt "a beautiful sunset" \
  --save-output

​

Notes

• --transformer-weights-path is the canonical flag for Nunchaku checkpoints. Older config names such as quantized_model_path are treated as compatibility aliases.
• Auto-detection only happens when the checkpoint basename matches svdq-(int4|fp4)_r{rank}.
• The CLI values are int4 and nvfp4. In filenames, the NVFP4 variant is written as fp4.
• Lightning checkpoints usually expect matching --num-inference-steps, such as 4 or 8.
• Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) or SM12x GPUs. Hopper (SM90) is currently rejected.

​

ModelSlim

• Installation
  # Clone repo and install msmodelslim:
  git clone https://gitcode.com/Ascend/msmodelslim.git
  cd msmodelslim
  bash install.sh
  
• Multimodal_sd quantization Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to Wan2.2-T2V-A14B to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card).

  Note: You can find pre-quantized validated models on modelscope/Eco-Tech.

  Run quantization using one-click quantization (recommended):
  msmodelslim quant \
    --model_path /path/to/wan2_2_float_weights \
    --save_path /path/to/wan2_2_quantized_weights \
    --device npu \
    --model_type Wan2_2 \
    --quant_type w8a8 \
    --trust_remote_code True
  
  For more detailed examples of quantization of models, as well as information about their support, see the examples section in ModelSLim repo.

  Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim.

• Auto-Detection and different formats For msmodelslim checkpoints, it’s enough to specify only --model-path, the detection of quantization occurs automatically for each layer using parsing of quant_model_description.json config. In the case of Wan2.2 only Diffusers weights storage format are supported, whereas modelslim saves the quantized model in the original Wan2.2 format, for conversion in use python/sglang/multimodal_gen/tools/wan_repack.py script:
  python wan_repack.py \
    --input-path {path_to_quantized_model} \
    --output-path {path_to_converted_model}
  
  After that, please copy all files from original Diffusers checkpoint (instead of transformer/tranfsormer_2 folders)
• Usage Example With auto-detected flow:
  sglang generate \
    --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \
    --prompt "a beautiful sunset" \
    --save-output
  
• Available Quantization Methods:
  • W4A4_DYNAMIC linear with online quantization of activations
  • W8A8 linear with offline quantization of activations
  • W8A8_DYNAMIC linear with online quantization of activations
  • mxfp8 linear in progress