> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # CLI reference > Run one-off generation tasks and launch the HTTP server from the command line. Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`. ### Overlay repos for non-diffusers models If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it through a self-hosted overlay repo. SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface. Override example: ```bash Command theme={null} export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{ "Wan-AI/Wan2.2-S2V-14B": { "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay", "overlay_revision": "main" } }' sglang generate \ --model-path Wan-AI/Wan2.2-S2V-14B \ --config configs/wan_s2v.yaml ``` The overlay repo should be a complete diffusers-style/componentized repo You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`. Notes: 1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for development and debugging. It accepts either a JSON object or a path to a JSON file, and can extend or replace built-in entries for the current process. 2. On the first load, SGLang will: * download overlay metadata from the overlay repo * download the required files from the original source repo * materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/` 3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory. ## Quick Start ### Generate ```bash Command theme={null} sglang generate \ --model-path Qwen/Qwen-Image \ --prompt "A beautiful sunset over the mountains" \ --save-output ``` ### Serve ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` For request and response examples, see [OpenAI-Compatible API](./openai_api). Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags. ## Common Options ### Model and runtime * `--model-path {MODEL}`: model path or Hugging Face model ID * `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`. * `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`. This is a server/load-time choice, unlike a request's `task`. * `--minimax-h3-adaln-cache-path {FILE}`: advanced MiniMax-H3-only inference cache. It replaces the checkpoint's AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant. * `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition. * `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter * `--lora-weight-name {FILE}`: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded. * `--lora-alpha {N}`: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and `adapter_config.json`. Do not set it when the adapter already records alpha metadata. * `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. * `--num-gpus {N}`: number of GPUs to use * `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. `speed` keeps `torch.compile` disabled unless a model-specific deployment config opts in after validation; pass `--enable-torch-compile true` to enable it explicitly. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. * `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Startup impact is model-dependent, so benchmark the target model before deployment. Disabled by default because checkpoint and model weights coexist temporarily, substantially increasing peak GPU memory. It is incompatible with DiT CPU/layerwise offload and FSDP. * `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both. * `--sp-degree {N}`: sequence parallelism size * `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only. * `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls * `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading. * `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism * `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). * `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic * `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency. * `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP. * `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. * `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse. * `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines * `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa` * `--attention-backend-config {CONFIG}`: attention backend configuration * `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](../models_with_ar). * `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server * `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server * `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it. * `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](../models_with_pe). ### Sampling and output * `--prompt {PROMPT}` and `--negative-prompt {PROMPT}` * `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation * `--num-inference-steps {STEPS}` and `--seed {SEED}` * `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`. * `--quality {lossless,high}`: request-level quality. `lossless` (default) keeps the exact reference path, bit-exact against the reference implementation; `high` opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific. * `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` * `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` For frame interpolation and upscaling, see [Post-Processing](./post_processing). ### Quantization For quantized transformer checkpoints, prefer: * `--model-path` for the base pipeline * `--transformer-path` for a quantized `transformers` transformer component folder * `--transformer-weights-path` for a quantized safetensors file, directory, or repo * `--quantization` for online quantization (apply quantization to unquantized models at load time, activations are quantized dynamically) * `--quantization-ignored-layers` layer name patterns to keep unquantized (e.g. `attention.to_`) Component checkpoint paths are selected separately, so changing DiT precision never silently changes prompt embeddings. For a native text encoder: * `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias * Quantization metadata is auto-detected from that checkpoint. Each native encoder must explicitly support the serialized format; this is not blanket quantization support for every component, and unsupported combinations fail before weight loading. For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}` compresses completed KV-cache chunks independently of transformer weight quantization. It is lossy and disabled by default. See [Realtime and Causal Video Models](../realtime_models) for the runtime and model scope, and [Quantization](../quantization) for supported quantization families and examples. ### Request logging * `--log-requests`: Log user-facing fields of all requests (default: `False`). The verbosity is decided by `--log-requests-level`. * `--log-requests-level {0|1|2|3}`: Verbosity level for request logging (default: `2`). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, ...). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt. * `--log-requests-format {text|json}`: Format for request logging (default: `text`). `text` is human-readable; `json` outputs structured JSON lines. * `--log-requests-target {TARGET...}`: Target(s) for request logging. Use `stdout` for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., `--log-requests-target stdout /my/log/dir`. ## Configuration Files Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file. ```bash Command theme={null} sglang generate --config config.yaml ``` Example: ```yaml Config theme={null} model_path: FastVideo/FastHunyuan-diffusers prompt: A beautiful woman in a red dress walking down a street output_path: outputs/ num_gpus: 2 sp_degree: 2 tp_size: 1 num_frames: 45 height: 720 width: 1280 num_inference_steps: 6 seed: 1024 fps: 24 precision: bf16 vae_precision: fp16 vae_tiling: true vae_sp: true enable_torch_compile: false ``` ## Generate `sglang generate` runs a single generation job and exits when the job finishes. ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --prompt "A curious raccoon" \ --save-output \ --output-path outputs \ --output-file-name "a-curious-raccoon.mp4" ``` HTTP server-only arguments are ignored by `sglang generate`. For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit). For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. ### Component Residency Use `--component-residency COMPONENT=MODE` to assign one runtime residency mode to each native pipeline component: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \ --prompt "A quiet city street after rain" ``` The available modes are: * `resident`: keep the complete component on the accelerator. * `component-offload`: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward. * `layerwise-offload`: keep component weights on CPU and stream its declared layers during execution. Selectors match exact loaded component keys from `model_index.json`, including names such as `transformer_2`, `audio_vae`, and `connectors`. The group selectors `dit`, `text_encoder`, `image_encoder`, and `vae` are also available, together with `all`. An exact key overrides a matching group, and a group overrides `all`. Components without a matching canonical selector retain their explicit legacy setting or automatic/model default. The existing `--dit-cpu-offload`, `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, and `--cpu-offload-components` options remain supported. New and legacy options may be mixed: `--component-residency` wins only for components it matches, while unmatched legacy settings remain effective. Legacy layerwise selectors take precedence over legacy component-offload selectors for the same component. Explicit `--dit-layerwise-offload false` makes the DiT resident unless another explicit DiT selector, such as `--dit-cpu-offload true` or `--component-residency dit=component-offload`, selects a different mode. Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`. ### Layerwise Offload Tuning Use layerwise offload when a component does not fit comfortably in GPU memory. The compatibility options `--dit-layerwise-offload` and `--layerwise-offload-components` remain available (`--layerwise-offload-modules` is an alias), while new deployments can select the mode directly: ```bash Command theme={null} sglang generate \ --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \ --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \ --dit-offload-prefetch-size 0 \ --prompt "A quiet city street after rain" ``` Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component. Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, and `--dit-layerwise-residency-policy` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. ## Serve `sglang serve` starts the HTTP server and keeps the model loaded for repeated requests. ```bash Command theme={null} sglang serve \ --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \ --text-encoder-cpu-offload \ --pin-cpu-memory \ --num-gpus 4 \ --ulysses-degree 2 \ --ring-degree 2 \ --port 30010 ``` ### Health endpoints SGLang Diffusion separates process liveness from inference readiness: | Endpoint | Success condition | Recommended use | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `GET /liveness` | The HTTP server is accepting requests. It remains `200` during server warmup. | Kubernetes liveness probe | | `GET /health` | The server is ready for normal inference traffic. It returns `503` while server-based synthetic warmup is running and `200` after it completes. | Startup and readiness probes | | `GET /health_generate` | Compatibility alias for `/health`. It does not currently issue a generation request in SGLang Diffusion. | Existing integrations only | `/health` gates only server-based warmup. With `--warmup-mode off` or `--warmup-mode request`, it returns `200` once the HTTP server starts; those modes do not promise that compilation or other first-request work has completed. If server-based warmup fails, the server terminates instead of reporting ready. Do not use `/health` as a liveness probe: a long server warmup can legitimately keep it at `503` for several minutes. ### Cloud Storage SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation. ```bash Command theme={null} export SGLANG_CLOUD_STORAGE_TYPE=s3 export SGLANG_S3_BUCKET_NAME=my-bucket export SGLANG_S3_ACCESS_KEY_ID=your-access-key export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key export SGLANG_S3_ENDPOINT_URL=https://minio.example.com ``` See [Environment Variables](../environment_variables) for the full set of storage options. ## Component Path Overrides Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `---path`. ```bash Command theme={null} sglang serve \ --model-path black-forest-labs/FLUX.2-dev \ --vae-path fal/FLUX.2-Tiny-AutoEncoder ``` The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory. ## Component Attention Backend Overrides Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`. ```bash Command theme={null} sglang generate \ --model-path Lightricks/LTX-2.3 \ --attention-backend fa \ --component-attention-backends text_encoder=torch_sdpa ``` The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` only while that component is being constructed. You can also pass dotted CLI entries: ```bash Command theme={null} sglang generate \ --model-path \ --component-attention-backends.text_encoder torch_sdpa \ --component-attention-backends.transformer fa ``` ## Diffusers Backend Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class. ### Key Options
Argument Values Description
--backend auto, sglang, diffusers Choose native SGLang, force native, or force diffusers
--attention-backend flash, \_flash\_3\_hub, sage, xformers, native Attention backend for diffusers pipelines
--trust-remote-code flag Required for models with custom pipeline classes
--vae-tiling and --vae-slicing flag Lower memory usage for VAE decode
--dit-precision and --vae-precision fp16, bf16, fp32 Precision controls
--enable-torch-compile flag Enable torch.compile
--cache-dit-config {PATH} Cache-DiT config for diffusers pipelines
### Example ```bash theme={null} sglang generate \ --model-path AIDC-AI/Ovis-Image-7B \ --backend diffusers \ --trust-remote-code \ --attention-backend flash \ --prompt "A serene Japanese garden with cherry blossoms" \ --height 1024 \ --width 1024 \ --num-inference-steps 30 \ --save-output \ --output-path outputs \ --output-file-name ovis_garden.png ``` For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.