CLI reference

​

Overlay repos for non-diffusers models

export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
  "Wan-AI/Wan2.2-S2V-14B": {
    "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
    "overlay_revision": "main"
  }
}'

sglang generate \
  --model-path Wan-AI/Wan2.2-S2V-14B \
  --config configs/wan_s2v.yaml

1. SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY is only an optional override for development and debugging. It accepts either a JSON object or a path to a JSON file, and can extend or replace built-in entries for the current process.
2. On the first load, SGLang will:
   • download overlay metadata from the overlay repo
   • download the required files from the original source repo
   • materialize a local standard component repo under ~/.cache/sgl_diffusion/materialized_models/
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.

​

Quick Start

​

Generate

sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output

​

Serve

sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010

​

Common Options

​

Model and runtime

• --model-path {MODEL}: model path or Hugging Face model ID
• --served-model-name {NAME}: stable model name exposed by serving APIs. Defaults to --model-id when set, otherwise --model-path.
• --model-variant {NAME}: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts fl2va and ref2va. This is a server/load-time choice, unlike a request’s task.
• --minimax-h3-adaln-cache-path {FILE}: advanced MiniMax-H3-only inference cache. It replaces the checkpoint’s AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant.
• --minimax-h3-adaln-online {true,false}: rebuild MiniMax-H3 AdaLN outputs from the checkpoint on demand instead of keeping the 24.2 GiB of adaln_proj weights resident. Works with any step count or schedule; requires the unquantized native-layout checkpoint. Built plans live in a GPU slab with per-plan LRU eviction and, by default, a pinned-host cache so previously seen schedules swap back in over PCIe instead of re-reading the checkpoint.
• --minimax-h3-adaln-plan-width {N}: widest timestep plan the online slab is sized for (default 4 covers every task; t2va needs 2, fl2va 3).
• --minimax-h3-adaln-host-cache-gb {GB}: pinned host memory per rank caching built AdaLN plans (default 8; 0 disables). A 50-step schedule needs about 0.9 (t2va) / 1.33 (fl2va) / 1.77 (ref2va) GB; over-capacity plan sets simply recompute. Expert escape hatches (GPU slot count, experimental fp32 rebuild) are the SGLANG_DIFFUSION_MINIMAX_H3_ADALN_* environment variables.
• --model-subfolder {PATH}: advanced direct override for a component subfolder inside the model repository. Prefer --model-variant when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition.
• --lora-path {PATH} and --lora-nickname {NAME}: load a LoRA adapter from a local path, Hugging Face repo/subfolder, or exact Hub file URL
• --lora-weight-name {FILE}: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded.
• --lora-alpha {N}: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and adapter_config.json. Do not set it when the adapter already records alpha metadata.
• PEFT adapter_config.json semantics are applied automatically for named adapter slots, RSLoRA, and per-layer alpha_pattern. Adapters that require unsupported auxiliary parameters or runtime behavior, such as DoRA, fail before weight injection instead of silently using ordinary LoRA math.
• --lora-merge-mode {auto|merge|dynamic}: choose how LoRA is applied. auto statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks.
• --num-gpus {N}: number of GPUs to use
• --performance-mode {manual|auto|speed|memory} / --mode: preset for latency/throughput and memory defaults. auto is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. speed keeps torch.compile disabled unless a model-specific deployment config opts in after validation; pass --enable-torch-compile true to enable it explicitly. Use manual to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
• --direct-gpu-weight-loading {true|false}: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Compatible tensors become parameter storage without another GPU copy, while tensors requiring transforms can still need temporary allocations. It is disabled by default; startup time and peak GPU memory remain model-dependent, so benchmark the target model before deployment. It is incompatible with DiT CPU/layerwise offload and FSDP.
• --tp-size {N}: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both.
• --sp-degree {N}: sequence parallelism size
• --dp-size {N} (alias --data-parallel-size): number of data-parallel replicas. Each replica is a full copy of the engine on num_gpus / N GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (num_gpus = dp × cfg × tp × sp); monolithic serving only.
• --ulysses-degree {N} and --ring-degree {N}: USP parallelism controls
• --kv-gather-degree {N}: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with --ulysses-degree/--ring-degree yet. When no SP degree is set explicitly, sp_degree=2 defaults to kv_gather_degree=2 (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
• --enable-cfg-parallel {true|false}: enable or explicitly disable CFG parallelism
• --encoder-parallel {auto|fold|dp|replicate}: how native encoders use the GPUs in each DiT replica. auto TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; fold shards native text/image encoders across the full replica whenever dimensions allow; dp splits a batched encode across supported native text encoder copies and composes with encoder TP; replicate disables folding and batch DP. Encoder collectives never cross --dp-size replicas. See Encoder Parallelism.
• --warmup-mode {off|request|server}: control startup warmup for sglang serve; off skips warmup, request primes the request path, and server runs a full synthetic server warmup before serving traffic
• --enable-torch-compile {true|false}: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
• --offload-during-compile {true|false}: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so max-autotune fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
• --enable-breakable-cuda-graph {true|false}: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires --warmup-resolutions for every served resolution because each resolution is captured separately. An extra-high or high request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only request-gated paths remain compatible.
• --bcg-text-buckets {N...}: prompt-length padding buckets for breakable CUDA graph capture/replay reuse.
• --attention-backend {BACKEND}: attention backend for native SGLang and diffusers pipelines
• --component-attention-backends {MAP}: per-component attention backend overrides, for example text_encoder=torch_sdpa,transformer=fa
• --attention-backend-config {CONFIG}: attention backend configuration
• --srt-encoder-url {HTTPADDRESS}: address of SGLang srt server with AR model for GLM-Image like models. See Models with AR Stage.
• --srt-encoder-timeout {SECONDS}: Timeout in seconds for HTTP requests to the SGLang encoder server
• --srt-encoder-connection-timeout {SECONDS}: TCP connection timeout in seconds for SGLang encoder server
• --scheduler-rpc-timeout {SECONDS}: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it.
• --pe-server-url {HTTPADDRESS}: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See Models with Prompt Enhancement.

​

Sampling and output

• --prompt {PROMPT} and --negative-prompt {PROMPT}
• --image-path {PATH} [{PATH} ...]: input image(s) for image-to-video or image-to-image generation
• --num-inference-steps {STEPS} and --seed {SEED}
• --num-outputs-per-prompt {N} / --num-outputs {N}: generate multiple outputs for each prompt. A scalar seed expands as seed + output_index.
• --quality {lossless,extra-high,high}: cumulative request-level optimization tier. lossless (default) keeps the selected deployment’s reference path and all unconditional bit-exact replacements. extra-high adds only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths. high includes the complete extra-high set and may also enable model-owned approximate optimizations. Separately configured quantization, attention, or caching options still apply. Support and validation constraints are model-specific.
• --height {HEIGHT}, --width {WIDTH}, --num-frames {N}, --fps {FPS}
• --output-path {PATH}, --output-file-name {NAME}, --save-output, --return-frames

​

Quantization

--component-paths.text_encoder COMPONENT_REPO_OR_DIRECTORY \
--component-weights-paths.text_encoder WEIGHTS_FILE_OR_REPO_FILE

• --component-paths.text_encoder {MODEL} replaces the text-encoder checkpoint; --text-encoder-path {MODEL} is its shorter alias
• --component-quantizations.text_encoder {METHOD} applies supported online quantization to an unquantized native encoder; pair it with --component-quantization-ignored-layers.text_encoder {PATTERN...} when selected layers must remain unquantized
• Quantization metadata is auto-detected from that checkpoint. The native loader accepts compatible serialized formats without a model-name allowlist and rejects implementations that do not construct the required quantized layers.

​

Request logging

• --log-requests: Log user-facing fields of all requests (default: False). The verbosity is decided by --log-requests-level.
• --log-requests-level {0|1|2|3}: Verbosity level for request logging (default: 2). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, …). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt.
• --log-requests-format {text|json}: Format for request logging (default: text). text is human-readable; json outputs structured JSON lines.
• --log-requests-target {TARGET...}: Target(s) for request logging. Use stdout for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., --log-requests-target stdout /my/log/dir.

​

Configuration Files

sglang generate --config config.yaml

model_path: FastVideo/FastHunyuan-diffusers
prompt: A beautiful woman in a red dress walking down a street
output_path: outputs/
num_gpus: 2
sp_degree: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: bf16
vae_precision: fp16
vae_tiling: true
vae_sp: true
enable_torch_compile: false

​

Generate

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --prompt "A curious raccoon" \
  --save-output \
  --output-path outputs \
  --output-file-name "a-curious-raccoon.mp4"

sglang serve --model-path meituan-longcat/LongCat-Image \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 1024x1024 \
  --enable-torch-compile false \
  --port 30010

sglang serve \
  --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 832x480 \
  --enable-torch-compile false \
  --port 30010

​

Component Residency

sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \
  --prompt "A quiet city street after rain"

• resident: keep the complete component on the accelerator.
• component-offload: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward.
• layerwise-offload: keep component weights on CPU and stream its declared layers during execution.

​

Layerwise Offload Tuning

sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \
  --dit-offload-prefetch-size 0 \
  --prompt "A quiet city street after rain"

sglang serve --model-path <MODEL> \
  --layerwise-offload-components dit,text_encoder \
  --layerwise-prefetch-size text_encoder=2 \
  --layerwise-resident-layers text_encoder=4 \
  --layerwise-residency-policy text_encoder=strided

• --layerwise-prefetch-size: how many layers to fetch ahead. Fractional values are a share of the stack, >= 1 an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers.
• --layerwise-resident-layers: how many layers stay on the GPU instead of being streamed. Resident layers are transferred once at startup, so they are removed from every pass. Fractional values are a share of the stack.
• --layerwise-residency-policy: leading keeps the first layers, strided spreads them across the stack so the transfers do not arrive as one burst.

​

Serve

sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010

​

Health endpoints

​

Cloud Storage

export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com

​

Component Path Overrides

sglang serve \
  --model-path black-forest-labs/FLUX.2-dev \
  --vae-path fal/FLUX.2-Tiny-AutoEncoder

​

Component Attention Backend Overrides

sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.text_encoder torch_sdpa \
  --component-attention-backends.transformer fa

​

Diffusers Backend

​

Key Options

​

Example

sglang generate \
  --model-path AIDC-AI/Ovis-Image-7B \
  --backend diffusers \
  --trust-remote-code \
  --attention-backend flash \
  --prompt "A serene Japanese garden with cherry blossoms" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 30 \
  --save-output \
  --output-path outputs \
  --output-file-name ovis_garden.png