Quantization on Ascend

• Dense model (e.g., Qwen3-14B-Q4_K_M.gguf):

python3 -m sglang.launch_server \
    --model-path Qwen3-14B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.7 --tp-size 2

• MoE model (e.g., Qwen3-30B-A3B-Q4_K_M.gguf):

python3 -m sglang.launch_server \
    --model-path Qwen3-30B-A3B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 2

Implementation Notes:

• GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
• MoE layers use npu_grouped_matmul and npu_moe_init_routing / npu_moe_finalize_routing for high-performance expert computation.
• TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.

• Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline MXFP8 quantization (msmodelslim pre-quantized weights, W8A8_MXFP8 scheme; no --quantization flag needed — auto-detected from quant_model_description.json):

python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

Implementation Notes:

• Online path: Fp8Config.get_quant_method() dispatches to NPUMXFP8LinearMethod. Weights are quantized once at load via npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn) and pre-transposed to [in, out]; activations are per-token quantized at inference and matmul runs via npu_quant_matmul(..., group_sizes=[1, 1, 32]) (block_size = 32).
• Offline path: ModelSlimMXFP8Scheme loads float8_e4m3fn weights + float8_e8m0fnu block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (.data assignment) — calling .contiguous() would physically reorder the pre-quantized layout and break the block-scale mapping.
• MoE MXFP8 (FusedMoE/TP) for LLMs is tracked separately and not part of this PR.

• Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp_w4a8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline W4A8 quantization (msmodelslim pre-quantized weights, W4A8_MXFP scheme; no --quantization flag needed — auto-detected from quant_model_description.json).

Implementation Notes:

• Weights are packed FP4 (float4_e2m1fn_x2, two nibbles per byte) with a UE8M0 per-block shared exponent (block_size = 32); activations are per-token MXFP8. Matmul runs via npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32]).
• The packed-FP4 dtype passed to the NPU ops (dst_type / x2_dtype / input_dtype) must be resolved from torch_npu.float4_e2m1fn_x2 (an int enum), not the torch.float4_e2m1fn_x2 dtype object, which recent op-plugin builds reject.
• Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration).

• Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time):

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp4 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1

• Offline W4A4 quantization (msmodelslim pre-quantized weights, W4A4_MXFP4 scheme; no --quantization flag needed — auto-detected from quant_model_description.json).

Implementation Notes:

• Online (NPUDualLevelMXFP4LinearMethod) uses dual-level MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via npu_dynamic_dual_level_mx_quant, and the matmul runs via npu_dual_level_quant_matmul (weight in FRACTAL_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS).
• Offline (ModelSlimMXFP4Scheme → NPUSingleLevelMXFP4OfflineLinearMethod) is single-level: msmodelslim’s W4A4_MXFP4 checkpoint ships single-level UE8M0 block scales (block_size = 32), so the matmul runs via npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32]). The online and offline paths therefore use different matmul kernels — they no longer share the matmul path.
• As with W4A8, the packed-FP4 dtype passed to the NPU ops (dst_type / x2_dtype) must be resolved from torch_npu.float4_e2m1fn_x2 (an int enum), not the torch.float4_e2m1fn_x2 dtype object, which recent op-plugin builds reject.
• Validated end-to-end on Ascend A5 hardware.

​

Diffusion Model Quantization on Ascend NPU

​

Online MXFP8 Quantization

# Start the diffusion server with online MXFP8 quantization
sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4

# One-shot generation
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a beautiful sunset over the mountains" \
  --save-output

​

Offline MXFP8 Quantization (ModelSlim)

msmodelslim quant \
  --model_path /path/to/wan2_2_float_weights \
  --save_path /path/to/wan2_2_mxfp8_weights \
  --device npu \
  --model_type Wan2_2 \
  --quant_type mxfp8 \
  --trust_remote_code True

Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.

python python/sglang/multimodal_gen/tools/wan_repack.py \
  --input-path /path/to/wan2_2_mxfp8_weights \
  --output-path /path/to/wan2_2_mxfp8_diffusers

sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset over the mountains" \
  --save-output