CLI reference

​

Overlay repos for non-diffusers models

export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
  "Wan-AI/Wan2.2-S2V-14B": {
    "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
    "overlay_revision": "main"
  }
}'

sglang generate \
  --model-path Wan-AI/Wan2.2-S2V-14B \
  --config configs/wan_s2v.yaml

1. SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY is only an optional override for development and debugging. It accepts either a JSON object or a path to a JSON file, and can extend or replace built-in entries for the current process.
2. On the first load, SGLang will:
   • download overlay metadata from the overlay repo
   • download the required files from the original source repo
   • materialize a local standard component repo under ~/.cache/sgl_diffusion/materialized_models/
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.

​

Quick Start

​

Generate

sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output

​

Serve

sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010

​

Common Options

​

Model and runtime

• --model-path {MODEL}: model path or Hugging Face model ID
• --served-model-name {NAME}: stable model name exposed by serving APIs. Defaults to --model-id when set, otherwise --model-path.
• --model-variant {NAME}: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts fl2va and ref2va. This is a server/load-time choice, unlike a request’s task.
• --minimax-h3-adaln-cache-path {FILE}: advanced MiniMax-H3-only inference cache. It replaces the checkpoint’s AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant.
• --model-subfolder {PATH}: advanced direct override for a component subfolder inside the model repository. Prefer --model-variant when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition.
• --lora-path {PATH} and --lora-nickname {NAME}: load a LoRA adapter
• --lora-weight-name {FILE}: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded.
• --lora-alpha {N}: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and adapter_config.json. Do not set it when the adapter already records alpha metadata.
• --lora-merge-mode {auto|merge|dynamic}: choose how LoRA is applied. auto statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
• --num-gpus {N}: number of GPUs to use
• --performance-mode {manual|auto|speed|memory} / --mode: preset for latency/throughput and memory defaults. auto is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. speed keeps torch.compile disabled unless a model-specific deployment config opts in after validation; pass --enable-torch-compile true to enable it explicitly. Use manual to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
• --direct-gpu-weight-loading {true|false}: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Startup impact is model-dependent, so benchmark the target model before deployment. Disabled by default because checkpoint and model weights coexist temporarily, substantially increasing peak GPU memory. It is incompatible with DiT CPU/layerwise offload and FSDP.
• --tp-size {N}: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both.
• --sp-degree {N}: sequence parallelism size
• --dp-size {N} (alias --data-parallel-size): number of data-parallel replicas. Each replica is a full copy of the engine on num_gpus / N GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (num_gpus = dp × cfg × tp × sp); monolithic serving only.
• --ulysses-degree {N} and --ring-degree {N}: USP parallelism controls
• --kv-gather-degree {N}: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with --ulysses-degree/--ring-degree yet. When no SP degree is set explicitly, sp_degree=2 defaults to kv_gather_degree=2 (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
• --enable-cfg-parallel {true|false}: enable or explicitly disable CFG parallelism
• --encoder-parallel {auto|fold|dp|replicate}: how text/image encoders use the GPUs in each DiT replica. auto TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; fold shards across the full replica whenever dimensions allow; dp splits a batched encode across encoder copies and composes with encoder TP; replicate disables folding and batch DP. Encoder collectives never cross --dp-size replicas. See Encoder Parallelism.
• --warmup-mode {off|request|server}: control startup warmup for sglang serve; off skips warmup, request primes the request path, and server runs a full synthetic server warmup before serving traffic
• --enable-torch-compile {true|false}: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
• --offload-during-compile {true|false}: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so max-autotune fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
• --enable-breakable-cuda-graph {true|false}: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires --warmup-resolutions for every served resolution because each resolution is captured separately.
• --bcg-text-buckets {N...}: prompt-length padding buckets for breakable CUDA graph capture/replay reuse.
• --attention-backend {BACKEND}: attention backend for native SGLang and diffusers pipelines
• --component-attention-backends {MAP}: per-component attention backend overrides, for example text_encoder=torch_sdpa,transformer=fa
• --attention-backend-config {CONFIG}: attention backend configuration
• --srt-encoder-url {HTTPADDRESS}: address of SGLang srt server with AR model for GLM-Image like models. See Models with AR Stage.
• --srt-encoder-timeout {SECONDS}: Timeout in seconds for HTTP requests to the SGLang encoder server
• --srt-encoder-connection-timeout {SECONDS}: TCP connection timeout in seconds for SGLang encoder server
• --scheduler-rpc-timeout {SECONDS}: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it.
• --pe-server-url {HTTPADDRESS}: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See Models with Prompt Enhancement.

​

Sampling and output

• --prompt {PROMPT} and --negative-prompt {PROMPT}
• --image-path {PATH} [{PATH} ...]: input image(s) for image-to-video or image-to-image generation
• --num-inference-steps {STEPS} and --seed {SEED}
• --num-outputs-per-prompt {N} / --num-outputs {N}: generate multiple outputs for each prompt. A scalar seed expands as seed + output_index.
• --quality {lossless,high}: request-level quality. lossless (default) keeps the exact reference path, bit-exact against the reference implementation; high opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific.
• --height {HEIGHT}, --width {WIDTH}, --num-frames {N}, --fps {FPS}
• --output-path {PATH}, --output-file-name {NAME}, --save-output, --return-frames

​

Quantization

• --model-path for the base pipeline
• --transformer-path for a quantized transformers transformer component folder
• --transformer-weights-path for a quantized safetensors file, directory, or repo
• --quantization for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically)
• --quantization-ignored-layers layer name patterns to keep unquantized (e.g. attention.to_)

• --component-paths.text_encoder {MODEL} replaces the text-encoder checkpoint; --text-encoder-path {MODEL} is its shorter alias
• Quantization metadata is auto-detected from that checkpoint. Each native encoder must explicitly support the serialized format; this is not blanket quantization support for every component, and unsupported combinations fail before weight loading.

​

Request logging

• --log-requests: Log user-facing fields of all requests (default: False). The verbosity is decided by --log-requests-level.
• --log-requests-level {0|1|2|3}: Verbosity level for request logging (default: 2). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, …). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt.
• --log-requests-format {text|json}: Format for request logging (default: text). text is human-readable; json outputs structured JSON lines.
• --log-requests-target {TARGET...}: Target(s) for request logging. Use stdout for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., --log-requests-target stdout /my/log/dir.

​

Configuration Files

sglang generate --config config.yaml

model_path: FastVideo/FastHunyuan-diffusers
prompt: A beautiful woman in a red dress walking down a street
output_path: outputs/
num_gpus: 2
sp_degree: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: bf16
vae_precision: fp16
vae_tiling: true
vae_sp: true
enable_torch_compile: false

​

Generate

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --prompt "A curious raccoon" \
  --save-output \
  --output-path outputs \
  --output-file-name "a-curious-raccoon.mp4"

​

Component Residency

sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \
  --prompt "A quiet city street after rain"

• resident: keep the complete component on the accelerator.
• component-offload: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward.
• layerwise-offload: keep component weights on CPU and stream its declared layers during execution.

​

Layerwise Offload Tuning

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \
  --dit-offload-prefetch-size 0 \
  --prompt "A quiet city street after rain"

​

Serve

sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010

​

Health endpoints

​

Cloud Storage

export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com

​

Component Path Overrides

sglang serve \
  --model-path black-forest-labs/FLUX.2-dev \
  --vae-path fal/FLUX.2-Tiny-AutoEncoder

​

Component Attention Backend Overrides

sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.text_encoder torch_sdpa \
  --component-attention-backends.transformer fa

​

Diffusers Backend

​

Key Options

​

Example

sglang generate \
  --model-path AIDC-AI/Ovis-Image-7B \
  --backend diffusers \
  --trust-remote-code \
  --attention-backend flash \
  --prompt "A serene Japanese garden with cherry blossoms" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 30 \
  --save-output \
  --output-path outputs \
  --output-file-name ovis_garden.png