python3 -m sglang.launch_server --help
Common launch commands
-
To use a configuration file, create a YAML file with your server arguments and specify it with
--config. CLI arguments will override config file values.Command -
To enable multi-GPU tensor parallelism, add
--tp 2. If it reports the error “peer access is not supported between these two devices”, add--enable-p2p-checkto the server launch command.Command -
To enable multi-GPU data parallelism, add
--dp 2. Data parallelism is better for throughput if there is enough memory. It can also be used together with tensor parallelism. The following command uses 4 GPUs in total. To run data-parallel replicas as separate servers behind a router, see DP, DPA and sgl-router.Command -
If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of
--mem-fraction-static.Command - See hyperparameter tuning on tuning hyperparameters for better performance.
-
For docker and Kubernetes runs, you need to set up shared memory which is used for communication between processes. See
--shm-sizefor docker and/dev/shmsize update for Kubernetes manifests. -
If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size.
Command
-
To enable fp8 weight quantization, add
--quantization fp8on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments. -
To enable fp8 kv cache quantization, add
--kv-cache-dtype fp8_e4m3or--kv-cache-dtype fp8_e5m2. -
To enable deterministic inference and batch invariant operations, add
--enable-deterministic-inference. More details can be found in deterministic inference document. -
If a multimodal server accepts requests from untrusted clients, restrict remote image, video, and audio URLs with
--allowed-media-domains. SGLang checks the initial URL and every redirect destination against the exact-hostname allowlist. Remote media downloads are limited to 64 MiB by default; adjust--media-url-max-file-size-mbwhen larger trusted media is required. Local media files named by a request are held to the same size limit; pass--no-allow-local-media-filesto reject them.WithoutCommand--allowed-media-domains, HTTP(S) media from any domain remains allowed for backward compatibility. Do not expose that configuration to untrusted users. Local paths anddata:URLs are not governed by the domain allowlist. -
To enable decode context parallelism for MLA models, add
--dcp-size N. See Decode Context Parallelism. -
If the model does not have a chat template in the Hugging Face tokenizer, you can specify a custom chat template. If the tokenizer has multiple named templates (e.g., ‘default’, ‘tool_use’), you can select one using
--hf-chat-template-name tool_use. -
To run tensor parallelism on multiple nodes, add
--nnodes 2. If you have two nodes with two GPUs on each node and want to run TP=4, letsgl-dev-0be the hostname of the first node and50000be an available port, you can use the following commands. If you meet deadlock, please try to add--disable-decode-cuda-graph --disable-prefill-cuda-graph -
(Note: This feature is out of maintenance and might cause error) To enable
torch.compileacceleration, add--enable-torch-compile. It accelerates small models on small batch sizes. By default, the cache path is located at/tmp/torchinductor_root, you can customize it using environment variableTORCHINDUCTOR_CACHE_DIR. For more details, please refer to PyTorch official documentation and Enabling cache for torch.compile.Command
Model and tokenizer
| Argument | Description | Defaults | Options |
|---|---|---|---|
--model-path--model | The path of the model weights. This can be a local folder or a Hugging Face repo ID. | None | Type: str |
--tokenizer-path | The path of the tokenizer. | None | Type: str |
--tokenizer-mode | Tokenizer mode. ‘auto’ will use the fast tokenizer if available, and ‘slow’ will always use the slow tokenizer. | auto | auto, slow |
--tokenizer-backend | Tokenizer backend. ‘huggingface’ uses the default HuggingFace tokenizers library; ‘fastokens’ uses the fastokens library for faster tokenization. Requires the fastokens package to be installed. | huggingface | huggingface, fastokens |
--tokenizer-worker-num | The worker num of the tokenizer manager. | 1 | Type: int |
--skip-tokenizer-init | If set, skip init tokenizer and pass input_ids in generate request. | False | bool flag (set to enable) |
--load-format | The format of the model weights to load. “auto” will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. “pt” will load the weights in the pytorch bin format. “safetensors” will load the weights in the safetensors format. “instanttensor” will load Safetensors weights using InstantTensor’s high-performance distributed loader. “npcache” will load the weights in pytorch format and store a numpy cache to speed up the loading. “dummy” will initialize the weights with random values, which is mainly for profiling.”gguf” will load the weights in the gguf format. “bitsandbytes” will load the weights using bitsandbytes quantization.”layered” loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. “flash_rl” will load the weights in flash_rl format. “fastsafetensors” and “private” are also supported. “runai_streamer” enables direct model loading from object storage and shared file systems. | auto | auto, pt, safetensors, instanttensor, npcache, dummy, sharded_state, gguf, bitsandbytes, mistral, layered, flash_rl, remote, remote_instance, fastsafetensors, private, runai_streamer |
--model-loader-extra-config | Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. | {} | Type: str |
--trust-remote-code | Whether or not to allow for custom models defined on the Hub in their own modeling files. | False | bool flag (set to enable) |
--context-length | The model’s maximum context length. Defaults to None (will use the value from the model’s config.json instead). | None | Type: int |
--is-embedding | Whether to use a CausalLM as an embedding model. | False | bool flag (set to enable) |
--disable-normalize-embedding | Disable the L2 normalization the embedding Pooler applies after pooling, returning the raw pooled vector. Only affects models whose Pooler is built with normalize=True; custom poolers (cross-encoder, BERT pooler, vision pooler) are unaffected. | False | bool flag (set to enable) |
--enable-multimodal | Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen. | None | bool flag (set to enable) |
--disable-multimodal | Disable the multimodal functionality to serve a multimodal model as text-only. Cannot be combined with --enable-multimodal. | False | bool flag (set to enable) |
--revision | The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | None | Type: str |
--model-impl | Which implementation of the model to use. “auto” will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. “sglang” will use the SGLang model implementation. “transformers” will use the Transformers model implementation. | auto | Type: str |
--detokenizer-worker-num | The worker num of the detokenizer manager. | 1 | Type: int |
--prefill-only-disable-kv-cache | Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with —is-embedding, —chunked-prefill-size=-1, —disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa_skip_kv_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page_size, head_num, head_dim) placeholders so GPU memory is not wasted. | False | bool flag (set to enable) |
--model-config-parser | Which model-config parser to use. “auto” picks “mistral” via the is_mistral_model name heuristic, else “hf” (AutoConfig over config.json). Plugins can register additional parsers via @register_model_config_parser. | auto | Type: str |
HTTP server
| Argument | Description | Defaults | Options |
|---|---|---|---|
--host | The host of the HTTP server. | 127.0.0.1 | Type: str |
--port | The port of the HTTP server. | 30000 | Type: int |
--fastapi-root-path | App is behind a path based routing proxy. | "" | Type: str |
--grpc-mode | Serve the encoder over gRPC on —port instead of HTTP. Requires —encoder-only. | False | bool flag (set to enable) |
--skip-server-warmup | If set, skip warmup. | False | bool flag (set to enable) |
--warmups | Specify custom warmup functions (csv) to run before server starts eg. —warmups=warmup_name1,warmup_name2 will run the functions warmup_name1 and warmup_name2 specified in warmup.py before the server starts listening for requests | None | Type: str |
--nccl-port | The port for NCCL distributed environment setup. Defaults to a random port. | None | Type: int |
--checkpoint-engine-wait-weights-before-ready | If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests. | False | bool flag (set to enable) |
--ssl-keyfile | The file path to the SSL key file. | None | Type: str |
--ssl-certfile | The file path to the SSL certificate file. | None | Type: str |
--ssl-ca-certs | The CA certificates file. | None | Type: str |
--ssl-keyfile-password | The password to decrypt the SSL keyfile. | None | Type: str |
--enable-ssl-refresh | Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires —ssl-certfile and —ssl-keyfile. | False | bool flag (set to enable) |
--enable-http-worker-reuseport | Give each Uvicorn HTTP worker its own SO_REUSEPORT listener. | False | bool flag (set to enable) |
--enable-http2 | Serve HTTP/2 alongside HTTP/1.1 on the same port: clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. The Python frontend switches from Uvicorn to Granian and requires ‘pip install sglang[http2]’; the Rust frontend (SGLANG_RUST_SERVER=1) serves it natively. | False | bool flag (set to enable) |
--http2-max-concurrent-streams | Maximum number of concurrent streams advertised on each HTTP/2 connection (1 to 2^32 - 1). Only applies with —enable-http2. | 200 | Type: int |
Quantization and data type
| Argument | Description | Defaults | Options |
|---|---|---|---|
--dtype | Data type for model weights and activations. * “auto” will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * “half” for FP16. Recommended for AWQ quantization. * “float16” is the same as “half”. * “bfloat16” for a balance between precision and range. * “float” is shorthand for FP32 precision. * “float32” for FP32 precision. | auto | auto, half, float16, bfloat16, float, float32 |
--quantization | The quantization method. | None | awq, fp8, mxfp8, gptq, marlin, gptq_marlin, awq_marlin, bitsandbytes, gguf, modelopt, modelopt_fp8, modelopt_fp4, nvfp4_online, modelopt_mixed, petit_nvfp4, w8a8_int8, w8a8_fp8, moe_wna16, w4afp8, mxfp4, auto-round, compressed-tensors, modelslim, quark, quark_int4fp8_moe, quark_mxfp4, mlx_q4, mlx_q8, unquant |
--quantization-param-path | Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. | None | Type: Optional[str] |
--kv-cache-dtype | Data type for kv cache storage. “auto” will use model data type. “bf16” or “bfloat16” for BF16 KV cache. “fp8_e5m2” and “fp8_e4m3” are supported for CUDA 11.8+. “mxfp8” is supported by the FA4 backend. “nvfp4” selects the NVFP4 FP4 E2M1 KV cache recipe; “fp4_mx_block16” selects the MX-style block-size-16 FP4 E2M1 KV cache recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+. Out-of-tree platform plugins can register further values for KV cache formats that only their KV pool implements; in-tree platforms reject them. | auto | auto, fp8_e5m2, fp8_e4m3, mxfp8, bf16, bfloat16, nvfp4, fp4_mx_block16, and values registered by platform plugins |
--indexer-kv-cache-dtype | Quantization recipe for the DSA lightning-indexer KV cache (Ascend NPU only). None disables the quantized indexer cache and keeps the default indexer behavior. “int8” stores the indexer keys in INT8 with FP16 scales to speed up the lightning indexer. | None | int8 |
--enable-fp32-lm-head | If set, the LM head outputs (logits) are in FP32. | False | bool flag (set to enable) |
--modelopt-quant | The ModelOpt quantization configuration. Supported values: ‘fp8’, ‘int4_awq’, ‘w4a8_awq’, ‘nvfp4’, ‘nvfp4_awq’. This requires the NVIDIA Model Optimizer library to be installed: pip install nvidia-modelopt | None | Type: str |
--modelopt-checkpoint-restore-path | Path to restore a previously saved ModelOpt quantized checkpoint. If provided, the quantization process will be skipped and the model will be loaded from this checkpoint. | None | Type: str |
--modelopt-checkpoint-save-path | Path to save the ModelOpt quantized checkpoint after quantization. This allows reusing the quantized model in future runs. | None | Type: str |
--modelopt-export-path | Path to export the quantized model in HuggingFace format after ModelOpt quantization. The exported model can then be used directly with SGLang for inference. If not provided, the model will not be exported. | None | Type: str |
--quantize-and-serve | Quantize the model with ModelOpt and immediately serve it without exporting. This is useful for development and prototyping. For production, it’s recommended to use separate quantization and deployment steps. | False | bool flag (set to enable) |
--rl-quant-profile | Path to the FlashRL quantization profile. Required when using —load-format flash_rl. | None | Type: str |
--enable-quant-communications | Enable INT8 quantization of TP communications (Supported only for NPU for Qwen3 series). | False | bool flag (set to enable) |
--enable-tf32-matmul | Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set_float32_matmul_precision). CUDA only. Automatically enabled for MiniMax-M2 and GLM-4 models. | False | bool flag (set to enable) |
Memory and scheduling
| Argument | Description | Defaults | Options |
|---|---|---|---|
--mem-fraction-static | The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. When unset, it is computed as (GPU memory - reserved memory) / GPU memory, defaulting to 0.88 if GPU memory cannot be detected. | None | Type: float |
--max-running-requests | The maximum number of running requests. | None | Type: int |
--max-queued-requests | The maximum number of queued requests. This option is ignored when using disaggregation-mode. | None | Type: int |
--max-total-tokens | The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. | None | Type: int |
--chunked-prefill-size | The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. | None | Type: int |
--prefill-decode-interval | The number of decode rounds to run after a prefill batch before scheduling the next prefill. By default, this is disabled except for profiled Qwen3-VL serving configurations on Hopper. Without data-parallel attention, the interval is skipped while no request is decoding. In data-parallel attention mode, the interval is synchronized across all DP ranks. Set to 0 to disable. | None | Type: int |
--prefill-decode-interval-min-bs | Apply --prefill-decode-interval only while the running batch holds at least this many requests. Below it, prefill is admitted every round so the batch fills quickly. In data-parallel attention mode, each rank compares its own running batch. Set to 0 to apply the interval at any batch size. | 0 | Type: int |
--prefill-max-requests | The maximum number of requests in a prefill batch. If not specified, there is no limit. | None | Type: int |
--enable-dynamic-chunking | Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks. | False | bool flag (set to enable) |
--max-prefill-tokens | The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model’s maximum context length. | 16384 | Type: int |
--schedule-policy | The scheduling policy of the requests. | fcfs | lpm, random, fcfs, dfs-weight, lof, priority, routing-key, uniboost |
--uniboost-gamma | Boost decay rate gamma (1/s) for --schedule-policy uniboost. Larger values behave more like fcfs, smaller values more like shortest-job-first. With --enable-uniboost-gamma-ada this is the starting value. Must be > 0. | 10.0 | Type: float |
--enable-uniboost-gamma-ada | Adapt --uniboost-gamma online to the observed latency tail (gamma-Ada): a heavy tail lowers gamma, a flat tail raises it. | False | bool flag (set to enable) |
--shortest-prefill-aging-factor | With —schedule-policy shortest-prefill-first, each prefill token processed while a request waits lowers its rank by this many tokens, bounding its wait to about its uncached length divided by the factor, in processed tokens. 0 disables aging. | 0.0 | Type: float |
--shortest-prefill-boundary-halving-chunks | With —schedule-policy shortest-prefill-first, scale the order and the prefill batches with the backlog B: the uncached waiting tokens plus the continuing chunk’s remainder. Requests with at least boundary = chunk * 2 ** (-max(0, B - decay_start) / (halving_chunks * chunk)) uncached tokens go in arrival order after the shorter ones and never join a continuing chunk, so they are not passed indefinitely; the —shortest-prefill-budget-* token budget caps each prefill batch. In units of the chunked prefill size per DP attention rank, so the policy keeps its shape when the chunk size changes. Requires chunked prefill. 0 disables. | 0.0 | Type: float |
--shortest-prefill-boundary-decay-start-chunks | With —shortest-prefill-boundary-halving-chunks, the backlog, in chunks, up to which the class boundary stays at the chunked prefill size; beyond it the boundary halves every halving_chunks. 0 halves from an empty queue. Unset: 2 * halving_chunks. | None | Type: float |
--shortest-prefill-budget-min-ratio | With —shortest-prefill-boundary-halving-chunks, cap each prefill batch at budget = min_ratio * chunk * 2 ** (B / (doubling_chunks * chunk)) prefill tokens, rounded down to the page and the chunk alignment; no cap once the budget reaches the chunked prefill size. Small batches on a short queue lower time to first token. A fraction of the chunked prefill size per DP attention rank, at most 1. 0 disables the budget. Unset: 1/8. | None | Type: float |
--shortest-prefill-budget-doubling-chunks | With the —shortest-prefill-budget-min-ratio budget, the backlog, in chunks, over which the budget doubles. Unset: halving_chunks. | None | Type: float |
--enable-priority-scheduling | Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. | False | bool flag (set to enable) |
--abort-on-priority-when-disabled | If set, abort requests that specify a priority when priority scheduling is disabled. | False | bool flag (set to enable) |
--schedule-low-priority-values-first | If specified with —enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. | False | bool flag (set to enable) |
--priority-scheduling-preemption-threshold | Minimum difference in priorities for an incoming request to have to preempt running request(s). | 10 | Type: int |
--schedule-conservativeness | How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. | 1.0 | Type: float |
--retraction-policy | The decode retraction policy to use when the KV cache is full. ‘length’ preserves the existing behavior and retracts short-output, long-input requests first. ‘priority’ retracts lower-priority requests first, using the same priority direction as priority scheduling. | length | length, priority |
--page-size | The number of tokens in a page. | 1 | Type: int |
--enable-page-major-kv-layout | Store the Mamba state and full/SWA KV caches as one entry per token slot, holding every layer’s state, in a single page-granular buffer instead of one buffer per layer. Implied by --enable-unified-memory, and currently supported only with it. | False | bool flag (set to enable) |
--enable-unified-memory | For hybrid Mamba/GDN and hybrid SWA models, replace the statically-partitioned pools (full-attention KV + SWA/Mamba conv/SSM state) with a single byte buffer split dynamically between the sub-pools, so KV-vs-state capacity flexes with the workload instead of being fixed at startup. Implies --enable-page-major-kv-layout. Requires stride-aware attention, linear-attention and Mamba backends; the allowed backends and the supported PD-disaggregation and speculative-decoding configurations are validated at startup. | False | bool flag (set to enable) |
--swa-full-tokens-ratio | The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. | 0.8 | Type: float |
--disable-hybrid-swa-memory | Disable the hybrid SWA memory. | False | bool flag (set to enable) |
--radix-eviction-policy | The eviction policy of radix trees. ‘lru’ stands for Least Recently Used, ‘lfu’ stands for Least Frequently Used. | lru | lru, lfu, slru, priority, tlru, joint_lru, state_cost |
--enable-prefill-delayer | Enable prefill delayer for DP attention to reduce idle time. | False | bool flag (set to enable) |
--prefill-delayer-max-delay-passes | Maximum forward passes to delay prefill. | 30 | Type: int |
--prefill-delayer-token-usage-low-watermark | Token usage low watermark for prefill delayer. | None | Type: float |
--prefill-delayer-queue-min-ratio | Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches min(running_req * ratio, max_prefill_bs) so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: 0.1–0.5. | None | Type: float |
--prefill-delayer-max-delay-ms | Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when --prefill-delayer-queue-min-ratio is set. Typical: 1000–5000. | None | Type: float |
--prefill-delayer-forward-passes-buckets | Custom buckets for prefill delayer forward passes histogram. 0 and max_delay_passes-1 will be auto-added. | None | List[float] |
--prefill-delayer-wait-seconds-buckets | Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added. | None | List[float] |
--disable-priority-preemption | Disable priority scheduling preemption. | False | bool flag (set to enable) |
--default-priority-value | Default priority for requests without explicit priority. | None | Type: int |
Runtime options
| Argument | Description | Defaults | Options |
|---|---|---|---|
--device | The device to use (‘cuda’, ‘xpu’, ‘hpu’, ‘npu’, ‘cpu’). Defaults to auto-detection if not specified. | None | Type: str |
--tensor-parallel-size--tp-size | The tensor parallelism size. | 1 | Type: int |
--pipeline-parallel-size--pp-size | The pipeline parallelism size. | 1 | Type: int |
--attention-context-parallel-size--attn-cp-size | The attention context parallelism size. | 1 | Type: int |
--dcp-size--decode-context-parallel-size | The decode context parallelism size. See Decode Context Parallelism. | 1 | Type: int |
--dcp-comm-backend | Communication backend for the DCP attention reduction: AllGather + ReduceScatter, one All-to-All exchange, or FlashInfer MNNVL All-to-All. Unset resolves to fi_a2a where the DCP group shares an MNNVL domain on Blackwell, else a2a on CUDA/ROCm/XPU, else ag_rs. | None | ag_rs, a2a, fi_a2a |
--dcp-replicate-q-proj | For MLA DCP with the a2a/fi_a2a backend: replicate the Q projection so each DCP rank computes the full-head query locally. Use —no-dcp-replicate-q-proj to disable the model-specific default. | None | bool flag (set to enable) |
--moe-data-parallel-size--moe-dp-size | The moe data parallelism size. | 1 | Type: int |
--pp-max-micro-batch-size | The maximum micro batch size in pipeline parallelism. | None | Type: int |
--pp-async-batch-depth | The async batch depth of pipeline parallelism. | 0 | Type: int |
--stream-interval | The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher | 1 | Type: int |
--incremental-streaming-output | Whether to output as a sequence of disjoint segments. | False | bool flag (set to enable) |
--random-seed | The random seed. | None | Type: int |
--mlx-enable-sampling | MLX backend only: sample decode tokens (temperature / top-k / top-p / min-p) instead of greedy argmax. Sampling runs inside the lazy MLX graph, so it works with the overlap scheduler; first tokens from prefill/extend are sampled too. Greedy requests keep exact argmax behavior. Also enables on the MLX path: grammar vocab masks and custom logit processors (these break decode chaining per step; custom processors run on pure-decode steps only), logit_bias, output logprobs (sampled token / top-k / token_ids; prompt input logprobs are not computed), NaN sanitization (SGLANG_SANITIZE_NAN_LOGITS), and per-request sampling_seed under —enable-deterministic-inference (deterministic within MLX only). Penalties are not applied. | False | bool flag (set to enable) |
--constrained-json-whitespace-pattern | (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to [\n\t ]* | None | Type: str |
--constrained-json-disable-any-whitespace | (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. | False | bool flag (set to enable) |
--watchdog-timeout | Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. | 300 | Type: float |
--soft-watchdog-timeout | Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. | None | Type: float |
--dist-timeout | Set timeout for torch.distributed initialization. | None | Type: int |
--download-dir | Model download directory for huggingface. | None | Type: str |
--model-checksum | Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID. | None | Type: str |
--base-gpu-id | The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. | 0 | Type: int |
--gpu-id-step | The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,… | 1 | Type: int |
--sleep-on-idle | Reduce CPU usage when sglang is idle. | False | bool flag (set to enable) |
--batch-notify-size | Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency. | 16 | Type: int |
--stream-response-default-include-usage | Include usage in every streaming response (even when stream_options is not specified). | False | bool flag (set to enable) |
--enable-streaming-session | Enable streaming session mode and StreamingSession wrapper. | False | bool flag (set to enable) |
--load-snapshot-publish-interval | Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately. | 15 | Type: int |
--use-ray | Use Ray actors for scheduler process management. | False | bool flag (set to enable) |
Logging
| Argument | Description | Defaults | Options |
|---|---|---|---|
--log-level | The logging level of all loggers. | info | Type: str |
--log-level-http | The logging level of HTTP server. If not set, reuse —log-level by default. | None | Type: str |
--log-requests | Log metadata, inputs, outputs of all requests. The verbosity is decided by —log-requests-level | False | bool flag (set to enable) |
--log-requests-level | 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. | 2 | 0, 1, 2, 3 |
--log-requests-format | Format for request logging: ‘text’ (human-readable) or ‘json’ (structured) | text | text, json |
--log-requests-target | Target(s) for request logging: ‘stdout’ and/or directory path(s) for file output. Can specify multiple targets, e.g., ‘—log-requests-target stdout /my/path’. | None | List[str] |
--uvicorn-access-log-exclude-prefixes | Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). | [] | List[str] |
--crash-dump-folder | Folder for crash diagnostics. Stores completed requests retained by the crash-dump buffer plus in-flight requests and, on NVIDIA CUDA, configures device coredumps under this folder. Existing CUDA coredump environment variables take precedence. If not specified, this option does not enable request crash dumping or set CUDA coredump defaults. | None | Type: str |
--enable-metrics | Enable log prometheus metrics. | False | bool flag (set to enable) |
--enable-mfu-metrics | Enable estimated MFU-related prometheus metrics. | False | bool flag (set to enable) |
--enable-metrics-for-all-schedulers | Enable —enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0. | False | bool flag (set to enable) |
--log-schedule-decisions | Log the outcome of every prefill pass that evaluates the waiting queue: picked vs left in queue, running and preempted counts, KV free/evictable tokens, and the picked rids (first 16). Logged on attention TP/CP rank 0 only. Passes that return early (e.g. running batch full, no free request slots) are not logged, and a pass that picks nothing is logged only when its counters differ from the previous empty pass. Verbose, so off by default. | False | bool flag (set to enable) |
--tokenizer-metrics-custom-labels-header | Specify the HTTP header for passing custom labels for tokenizer metrics. | x-custom-labels | Type: str |
--tokenizer-metrics-allowed-custom-labels | The custom labels allowed for tokenizer metrics. The labels are specified via a dict in ‘—tokenizer-metrics-custom-labels-header’ field in HTTP requests, e.g., {‘label1’: ‘value1’, ‘label2’: ‘value2’} is allowed if ‘—tokenizer-metrics-allowed-custom-labels label1 label2’ is set. | None | List[str] |
--bucket-time-to-first-token | The buckets of time to first token, specified as a list of floats. | None | List[float] |
--bucket-inter-token-latency | The buckets of inter-token latency, specified as a list of floats. | None | List[float] |
--bucket-e2e-request-latency | The buckets of end-to-end request latency, specified as a list of floats. | None | List[float] |
--prompt-tokens-buckets | The buckets rule of prompt tokens. Supports 3 rule types: ‘default’ uses predefined buckets; ‘tse <middle> <base> <count>’ generates two sides exponential distributed buckets (e.g., ‘tse 1000 2 8’ generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); ‘custom <value1> <value2> …’ uses custom bucket values (e.g., ‘custom 10 50 100 500’). | None | List[str] |
--generation-tokens-buckets | The buckets rule for generation tokens histogram. Supports 3 rule types: ‘default’ uses predefined buckets; ‘tse <middle> <base> <count>’ generates two sides exponential distributed buckets (e.g., ‘tse 1000 2 8’ generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); ‘custom <value1> <value2> …’ uses custom bucket values (e.g., ‘custom 10 50 100 500’). | None | List[str] |
--gc-warning-threshold-secs | The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. | 0.0 | Type: float |
--decode-log-interval | The log interval of decode batch. | 40 | Type: int |
--enable-request-time-stats-logging | Enable per request time stats logging | False | bool flag (set to enable) |
--kv-events-config | Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. Runtime-load publishing for load-aware routers is a separate opt-in; see --load-publish-endpoint. Set replay_endpoint (e.g. tcp://*:5558) to let routers re-fetch dropped batches; /server_info advertises its port. | None | Type: str |
--load-publish-endpoint | Opt in to the runtime-load PUB socket that load-aware routers subscribe to. Off by default (unset or off). Use auto to reserve the dp_size ports packed after the --kv-events-config range, or a wildcard-host TCP address (e.g. tcp://*:6000) to place it explicitly; rank r binds port+r and /server_info advertises the base under the kv_events block. Requires --kv-events-config; startup fails if set without one, not bindable, or overlapping the KV range. auto reserves 2*dp_size ports from the KV base — space co-hosted engines accordingly. The router-facing update cadence follows --load-snapshot-publish-interval, so a large value there also staleness-caps this feed. | None | Type: str |
--enable-trace | Enable opentelemetry trace | False | bool flag (set to enable) |
--trace-modules | Select the components to trace. Available options are ‘request’ and ‘mooncake’. Format: <module1 name>,<module2 name>,… | request | Type: str |
--otlp-traces-endpoint | Config opentelemetry collector endpoint if —enable-trace is set. format: <ip>:<port> | localhost:4317 | Type: str |
--extra-metric-labels | The custom labels for metrics. e.g. ’{“label1”: “value1”, “label2”: “value2”}‘ | None | Type: str |
--enable-forward-pass-metrics | Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server_args.forward_pass_metrics_ipc_name. | False | bool flag (set to enable) |
--forward-pass-metrics-worker-id | — | "" | Type: str |
--forward-pass-metrics-ipc-name | — | None | Type: str |
RequestMetricsExporter configuration
| Argument | Description | Defaults | Options |
|---|---|---|---|
--export-metrics-to-file | Export performance metrics for each request to local file (e.g. for forwarding to external systems). | False | bool flag (set to enable) |
--export-metrics-to-file-dir | Directory path for writing performance metrics files (required when —export-metrics-to-file is enabled). | None | Type: str |
Responses storage
Responses storage is now disabled by default, including on standalone servers. Add--enable-response-store to retain responses for retrieval, previous_response_id chaining, and background requests. Storage is process-local and in-memory, has no TTL or size limit, and is lost on restart.
Foreground generation and streaming still accept the API’s default store: true without retaining responses or message history when server storage is disabled. The response’s store field echoes the client request; the server flag controls persistence. With storage enabled, store: false opts out for the current request while allowing it to read a stored predecessor.
Background requests, including streaming requests, require both --enable-response-store and store: true. Without the flag, chaining, background requests, retrieval, and cancellation return HTTP 400. Cancellation applies to detached background requests; active streams have no stored response until completion.
Combining --enable-response-store with --disaggregation-mode=prefill or decode fails at startup. PD clients must send explicit conversation history in foreground requests. Built-in web search and code interpreter calls are also unsupported under PD.
API related
| Argument | Description | Defaults | Options |
|---|---|---|---|
--api-key | Set API key of the server. It is also used in the OpenAI API compatible server. | None | Type: str |
--admin-api-key | Set admin API key (sent as Authorization: Bearer <admin_api_key>). Sensitive control endpoints (e.g., weights update, LoRA load/unload, profiling, /pause_generation, /hicache/storage-backend) always require it and return HTTP 403 when it is unset, lighter ones (e.g., /flush_cache, /abort_request) require it only once it is set (else —api-key if set), and all other endpoints (inference, /server_info) only check —api-key. Defaults to SGLANG_ADMIN_API_KEY when the flag is not given. | None | Type: str |
--served-model-name | Override the model name returned by the v1/models endpoint in OpenAI API server. | None | Type: str |
--weight-version | Version identifier for the model weights. Defaults to ‘default’ if not specified. | default | Type: str |
--enable-prefill-weight-versions | Return the weight version that computed each prompt token’s KV cache in meta_info[‘prefill_weight_versions’]. | False | Flag |
--chat-template | The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. | None | Type: str |
--hf-chat-template-name | When the HuggingFace tokenizer has multiple chat templates (e.g., ‘default’, ‘tool_use’, ‘rag’), specify which named template to use. If not set, the first available template is used. | None | Type: str |
--default-chat-template-kwargs | A JSON object used as defaults for chat_template_kwargs by Chat Completions, Responses, Anthropic-compatible requests, and chat tokenization. Direct Ollama and embedding paths do not consume these defaults. For ordinary template keys, explicit per-request values take precedence. Keys are specific to the selected model’s chat template; reasoning_effort is normalized separately. | None | Type: JSON / Dict |
--completion-template | The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | None | Type: str |
--file-storage-path | The path of the file storage in backend. | sglang_storage | Type: str |
--enable-cache-report | Return number of cached tokens in usage.prompt_tokens_details for each openai request. | False | bool flag (set to enable) |
--reasoning-parser | Specify the parser for reasoning models. Use auto to detect the parser from the model’s chat template. | None | auto, apertus2509, deepseek-r1, deepseek-v3, deepseek-v4, dots, ernie55, glm45, ling3, hunyuan, gpt-oss, k2_horizon, kimi, kimi_k2, kimi_k3, mimo, muse, poolside_v1, qwen3, qwen3-thinking, minimax, minimax-append-think, minimax-m3, step3, step3p5, mistral, nemotron_3, interns1, gemma4, gigachat35, inkling, cohere_command4 |
--tool-call-parser | Specify the parser for handling tool-call interactions. Use auto to detect the parser from the model’s chat template. | None | auto, apertus2509, cohere_command4, deepseekv3, deepseekv31, deepseekv32, deepseekv4, dots, ernie55, glm, glm47, gpt-oss, k2_horizon, kimi_k2, kimi_k3, lfm2, ling3, llama3, mimo, minicpm5, mistral, muse, poolside_v1, pythonic, qwen, qwen3_coder, spark25, step3, step3p5, minimax-m2, minimax-m3, trinity, interns1, hermes, hunyuan, gigachat3, gigachat35, gemma4, inkling |
--tool-server | Either ‘demo’ or a comma-separated list of tool server urls to use for the model. If not specified, no external tool server will be used. Native GPT-OSS web_search can still be enabled with EXA_API_KEY. | None | Type: str |
--sampling-defaults | Where to get default sampling parameters. ‘openai’ uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). ‘model’ uses the model’s generation_config.json to get the recommended sampling parameters if available. Default is ‘model’. | model | openai, model |
--strip-thinking-cache | Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents. | False | bool flag (set to enable) |
--enable-strict-thinking | Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering. See Thinking Budget for per-request thinking caps. | False | bool flag (set to enable) |
--asr-max-buffer-seconds | Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s. | 60 | Type: int |
--asr-max-concurrent-sessions | Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error{code:too_many_sessions} frame, and closed. Default 32. | 32 | Type: int |
Data parallelism
| Argument | Description | Defaults | Options |
|---|---|---|---|
--data-parallel-size--dp-size | The number of data-parallel replicas of the model. | 1 | Type: int |
--attention-data-parallel-size--attn-dp-size | The attention data parallelism size: the number of data-parallel attention groups inside the TP group, one scheduler each, while the FFN stays tensor parallel. | 1 | Type: int |
--load-balance-method | The load balancing strategy for data parallelism. The total_tokens algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. | auto | auto, round_robin, follow_bootstrap_room, total_requests, total_tokens |
Multi-node distributed serving
| Argument | Description | Defaults | Options |
|---|---|---|---|
--dist-init-addr--nccl-init-addr | The host address for initializing distributed backend (e.g., 192.168.0.2:25000). | None | Type: str |
--nnodes | The number of nodes. | 1 | Type: int |
--node-rank | The node rank. | 0 | Type: int |
Model override args
| Argument | Description | Defaults | Options |
|---|---|---|---|
--json-model-override-args | A dictionary in JSON string format used to override default model configurations. | {} | Type: str |
--preferred-sampling-params | json-formatted sampling settings that will be returned in /get_model_info | None | Type: str |
LoRA
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-lora | Enable LoRA support for the model. This argument is automatically set to True if --lora-paths is provided for backward compatibility. | False | Bool flag (set to enable) |
--enable-lora-overlap-loading | Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. | False | Bool flag (set to enable) |
--max-lora-rank | The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in —lora-paths. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. | None | Type: int |
--lora-target-modules | The union set of all target modules where LoRA should be applied (e.g., q_proj, k_proj, gate_proj). If not specified, it will be automatically inferred from the adapters provided in —lora-paths. You can also set it to all to enable LoRA for all supported modules; note this may introduce minor performance overhead. | None | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, qkv_proj, gate_up_proj, all |
--lora-paths | The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {“lora_name”: str, “lora_path”: str, “pinned”: bool}. | None | Type: List[str] / JSON objects |
--max-loras-per-batch | Maximum number of adapters for a running batch, including base-only requests. | 8 | Type: int |
--max-loaded-loras | If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ —max-loras-per-batch. | None | Type: int |
--lora-eviction-policy | LoRA adapter eviction policy when the GPU memory pool is full. | lru | lru, fifo |
--lora-backend | Choose the kernel backend for multi-LoRA serving. | csgmv | triton, csgmv, ascend, torch_native |
--max-lora-chunk-size | Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when —lora-backend is csgmv. Larger values may improve performance. | 16 | 16, 32, 64, 128 |
--lora-drain-wait-threshold | For LoRA adapters whose requests wait longer than this threshold (seconds) and need a slot, selectively drain unpinned running adapters to make room. Draining is best effort: freed slots are not reserved and waiting time is not bounded. Set to 0 to disable draining (default). | 0.0 | Type: float |
--experts-shared-outer-loras | Force shared outer LoRA mode for MoE models. When set, w1/w3 lora_A and w2 lora_B are shared across experts (expert_dim=1). Use —no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights. | None | Type: str |
--lora-use-virtual-experts | Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation. | False | bool flag (set to enable) |
--lora-strict-loading | Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error. | False | Type: str |
Kernel Backends (Attention, Sampling, Grammar, GEMM)
| Argument | Description | Defaults | Options |
|---|---|---|---|
--attention-backend | Choose the kernels for attention layers. | None | triton, torch_native, flex_attention, dsa, dsv4, ernie55, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu |
--prefill-attention-backend | Choose the kernels for prefill attention layers (have priority over —attention-backend). | None | triton, torch_native, flex_attention, dsa, dsv4, ernie55, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu |
--decode-attention-backend | Choose the kernels for decode attention layers (have priority over —attention-backend). | None | triton, torch_native, flex_attention, dsa, dsv4, ernie55, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu |
--enable-nvidia-megakernel | Enable the NVIDIA megakernel from the sgl-nvidia-mega package, or for MiniMax-M3.1 from the SGLang plugin of the MSA package. Supported decode and verify steps use its fused kernels; other steps use the standard forward. An unsupported startup configuration raises an error listing the reasons. Experimental feature. | False | bool flag (set to enable) |
--enable-amd-monokernel | Enable the AMD monokernels from the sgl-amd-mono package. Supported decode and verify steps use persistent kernels; other steps use the standard forward. An unsupported startup configuration raises an error listing the reasons. Experimental feature. | False | bool flag (set to enable) |
--sampling-backend | Choose the kernels for sampling layers. | None | flashinfer, pytorch, ascend |
--grammar-backend | Choose the backend for grammar-guided decoding. | None | xgrammar, outlines, llguidance, none |
--mm-attention-backend | Set multimodal attention backend. | None | sdpa, fa3, fa4, triton_attn, ascend_attn, aiter_attn, flashinfer_cudnn, amx_attn, xpu_attn |
--dsa-prefill-backend | DSA backend for the prefill stage (overrides --attention-backend when running DeepSeek DSA-style attention). | Auto (hardware-dependent) | flashmla_sparse, flashmla_sparse_q8, triton_sparse_mla, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm, prims_ts |
--dsa-triton-union | Union group size for --dsa-prefill-backend triton_sparse_mla: adjacent query tokens share one gathered index set (exact; an ownership mask restores each token’s own softmax). Requires num_q_heads * union to be a power of two in [16, 32]; forced off under --enable-deterministic-inference. | 0 | 0, 2, 4 |
--dsa-decode-backend | DSA backend for the decode stage when running DeepSeek DSA-style attention. Overrides --attention-backend for decoding. | Auto (hardware-dependent) | flashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm, prims_ts |
--dsv4-prefill-backend | DeepSeek-V4 sparse prefill backend. The experimental cutedsl_h16 path is specialized for BF16, 16 TP-local query heads, and SM90; speculative, CUDA-graph, TBO, CP and bounded-replay forwards retain their existing paths. DeepSeek-V4.1 is unsupported. Its first eligible shape triggers JIT compilation. Padded combined TOPK is limited to 16384, supporting the default 1M context. The existing sparse-prefill threshold and environment selection still apply. | auto | auto, flashmla_sparse, flashmla_sparse_q8, cutedsl_h16 |
--enable-flashmla-prefill-token-packing | Pack adjacent Kimi-K3.1 prefill tokens into one FlashMLA query tile. Requires SM100/SM103, FP8 V32 KV cache and at most 32 heads per TP rank; disabled during deterministic inference. On by default for Kimi-K3.1; --no-enable-flashmla-prefill-token-packing disables it. | None | Type: bool |
--dsa-topk-backend | Choose the DSA indexer top-k backend for the target model. The torch backend currently requires SGLANG_DSA_FUSE_TOPK=false. | sgl-kernel | sgl-kernel, torch, flashinfer |
--enable-deepseek-v4-fp4-indexer | Enable the experimental FP4 C4 indexer path for DeepSeek V4. When unset, SGLang keeps the default DeepSeek V4 indexer path. Requires SM100 GPUs with DeepGEMM FP4 indexer support. | False | bool flag (set to enable) |
--enable-fp4-indexer | Use FP4 queries and index-K cache in the DSA lightning indexer of DeepSeek-V3.2-style sparse attention, ERNIE 5.5 Flash’s MLA layers included; requires SM100. DeepSeek V4’s C4 indexer has its own switch, --enable-deepseek-v4-fp4-indexer. Default keeps the FP8 indexer. | False | bool flag (set to enable) |
--enable-dsa-fp4-indexer | Store the DSA indexer keys in MXFP4 and score them with AITER’s FP4 MQA-logits kernels (Kimi-K3.1 on gfx950 with DCP): the row-group kernel in decode and target verify, the prefill kernel over the DCP-gathered keys in prefill. | False | bool flag (set to enable) |
--fp8-gemm-backend | Choose the runner backend for Blockwise FP8 GEMM operations. For MXFP8 dense GEMM, auto selects flashinfer_cutedsl when FlashInfer reports support (currently SM100/SM103), and otherwise selects flashinfer_cutlass on supported Blackwell GPUs. Options also include ‘deep_gemm’ (JIT-compiled), ‘flashinfer_trtllm’ (FlashInfer TRTLLM backend; SM100/SM103 only), ‘flashinfer_cutlass’ (FlashInfer CUTLASS backend), ‘flashinfer_cutedsl’ (FlashInfer CuTe DSL MXFP8 backend; SM100/SM103 only), ‘flashinfer_deepgemm’ (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), ‘cutlass’ (optimal for SM120 GPUs), ‘triton’ (fallback, widely compatible), ‘aiter’ (ROCm only), and ‘halops’ (MXFP8 w8a8 dense GEMM from the optional HALOps wheel; SM100 only). | auto | auto, deep_gemm, flashinfer_trtllm, flashinfer_cutlass, flashinfer_cutedsl, flashinfer_deepgemm, cutlass, triton, aiter, halops |
--fp4-gemm-backend | Choose the runner backend for NVFP4 GEMM operations. Options: ‘auto’ (default; selects flashinfer_cutedsl on SM100, marlin on SM80-SM90, flashinfer_cutlass otherwise (including SM120)), ‘flashinfer_cutlass’ (FlashInfer CUTLASS backend), ‘flashinfer_cudnn’ (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), ‘flashinfer_cutedsl’ (FlashInfer CuTe DSL backend), ‘flashinfer_trtllm’ (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), ‘marlin’ (weight-only W4A16 fallback for SM80-SM90). Requires FlashInfer to be installed. | auto | auto, flashinfer_cudnn, flashinfer_cutedsl, flashinfer_cutlass, flashinfer_trtllm, marlin |
--bf16-gemm-backend | Choose the backend for unquantized BF16 GEMM operations. Options: auto (default; selects cutedsl on SM10x GPUs, except deterministic inference selects torch; otherwise uses cuBLAS via torch.nn.functional.linear), cutedsl (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the CuTe DSL kernel and cuBLAS), torch (always uses cuBLAS via torch.nn.functional.linear). | auto | auto, cutedsl, torch |
--disable-flashinfer-autotune | Flashinfer autotune is enabled by default. Set this flag to disable the autotune. | False | bool flag (set to enable) |
--flashinfer-autotune-skip-ops | FlashInfer custom-op identifiers to skip during autotuning. See FlashInfer’s autotuning documentation. Skipped ops use the heuristic fallback. SGLang temporarily skips mxfp8_gemm by default due to an IMA. | None | string |
--radix-cache-backend | Name of a radix-cache backend previously registered via register_radix_cache_backend. Omit this flag to use the built-in default cache selection chain. | None | Type: str |
Speculative decoding
| Argument | Description | Defaults | Options |
|---|---|---|---|
--speculative-algorithm | Speculative algorithm. | None | EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM |
--speculative-draft-model-path--speculative-draft-model | The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. | None | Type: str |
speculative_boundary_reductionServer args field; no CLI flag | Resolved FFN boundary reduction policy for the speculative draft. Argument processing derives this field from boundary_reduction and the draft model configuration; an independent value is overwritten by resolution. See FFN boundary reduction. | rs+rsv before resolution; model-dependent afterward | ar, rs, rsv, rs+rsv |
--speculative-draft-model-revision | The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | None | Type: str |
--speculative-draft-load-format | The format of the draft model weights to load. If not specified, will use the same format as --load-format. Use ‘dummy’ to initialize draft model weights with random values for profiling. | None | auto, pt, safetensors, instanttensor, npcache, dummy, sharded_state, gguf, bitsandbytes, mistral, layered, flash_rl, remote, remote_instance, fastsafetensors, private, runai_streamer |
--speculative-num-steps | The number of steps sampled from draft model in Speculative Decoding. | None | Type: int |
--speculative-eagle-topk | The number of tokens sampled from the draft model in eagle2 each step. | None | Type: int |
--speculative-num-draft-tokens | The number of tokens sampled from the draft model in Speculative Decoding. | None | Type: int |
--speculative-accept-threshold-single | Accept a draft token if its probability in the target model is greater than this threshold. | 1.0 | Type: float |
--speculative-accept-threshold-acc | The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc). | 1.0 | Type: float |
--speculative-use-block-verification | Use block verification for EAGLE/EAGLE3/NEXTN on CUDA or ROCm (requires topk=1). | False | Type: bool |
--speculative-token-map | The path of the draft model’s small vocab table. | None | Type: str |
--speculative-attention-mode | Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of ‘prefill’ (default) or ‘decode’. | prefill | prefill, decode |
--speculative-draft-attention-backend | Attention backend for speculative decoding drafting. | None | Same as attention backend options |
--speculative-dsa-topk-backend | Choose the DSA indexer top-k backend for speculative draft workers. The torch backend currently requires SGLANG_DSA_FUSE_TOPK=false. | sgl-kernel | sgl-kernel, torch, flashinfer |
--speculative-dspark-lm-head-dtype | DSPARK only. Quantized dtype for the draft’s projection through the target lm_head; the weight is quantized once and cached. fp8 requires gfx95x and mxfp8 requires Blackwell. Target verify logits are unaffected. Keeps the lm_head’s own dtype if unset. | None | fp8, mxfp8 |
--speculative-moe-runner-backend | MOE backend for EAGLE speculative decoding, see --moe-runner-backend for options. Same as moe runner backend if unset. | None | auto, deep_gemm, triton, flashinfer_trtllm, experimental_sgl_trtllm, flashinfer_trtllm_routed, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, aiter, marlin |
--speculative-moe-a2a-backend | MOE A2A backend for EAGLE speculative decoding, see --moe-a2a-backend for options. Same as moe a2a backend if unset. The experimental ascend_ops backend is supported on Ascend NPU and uses --moe-ascend-ops-config. | None | none, deepep, mooncake, nixl, mori, ascend_fuseep, flashinfer_nvlink_one_sided, megamoe, pplx, ascend_ops |
--speculative-enable-w4a4-mxfp4-megamoe | Whether the draft model’s MXFP4 MegaMoE layers use the W4A4 mxf4xmxf4 MMA type. Pass --no-speculative-enable-w4a4-mxfp4-megamoe to keep the draft on fp8xfp4 (W4A8) while the target runs W4A4. Same as --enable-w4a4-mxfp4-megamoe if unset. | None | Type: bool |
--speculative-draft-model-quantization | The quantization method for speculative model. | None | Same as --quantization options |
--speculative-dflash-block-size | DFLASH only. Block size (verify window length). Alias of —speculative-num-draft-tokens for DFLASH. | None | Type: int |
--speculative-draft-window-size | Sliding window size for the draft model. Honored by Llama EAGLE-3 (LlamaForCausalLMEagle3) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context. | None | Type: int |
Ngram speculative decoding
| Argument | Description | Defaults | Options |
|---|---|---|---|
--speculative-ngram-min-bfs-breadth | The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | 1 | Type: int |
--speculative-ngram-max-bfs-breadth | The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | 10 | Type: int |
--speculative-ngram-match-type | Ngram tree-building mode. BFS selects recency-based expansion and PROB selects frequency-based expansion. This setting is forwarded to the ngram cache implementation. | BFS | BFS, PROB |
--speculative-ngram-max-trie-depth | Maximum suffix length stored and matched by the ngram trie. | 18 | Type: int |
--speculative-ngram-capacity | The cache capacity for ngram speculative decoding. | 10000000 | Type: int |
--speculative-ngram-external-corpus-path | Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add_external_corpus. | None | Type: str |
--speculative-ngram-external-sam-budget | Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding. | 0 | Type: int |
--speculative-ngram-external-corpus-max-tokens | Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget. | 10000000 | Type: int |
--speculative-adaptive | Enable adaptive speculative decoding that dynamically adjusts num_steps based on acceptance rate. | False | bool flag (set to enable) |
--speculative-adaptive-config | Path to a JSON config file for adaptive speculative decoding tuning knobs. | None | Type: str |
Multi-layer Eagle speculative decoding
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-multi-layer-eagle | Enable multi-layer Eagle speculative decoding. | False | bool flag (set to enable) |
MoE
| Argument | Description | Defaults | Options |
|---|---|---|---|
--expert-parallel-size--ep-size--ep | The expert parallelism size. | 1 | Type: int |
--moe-a2a-backend | Select the backend for all-to-all communication for expert parallelism. flashinfer_nvlink_one_sided is FlashInfer’s NVLink one-sided all-to-all. | none | none, deepep, mooncake, nixl, mori, ascend_fuseep, flashinfer_nvlink_one_sided, megamoe, pplx, ascend_ops |
--moe-runner-backend | Choose the runner backend for MoE. | auto | auto, deep_gemm, triton, flashinfer_trtllm, experimental_sgl_trtllm, flashinfer_trtllm_routed, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, aiter, marlin |
--flashinfer-mxfp4-moe-precision | Choose the computation precision of flashinfer mxfp4 moe. On SM90, fp8 selects the Humming-style MXFP4-weight x FP8-activation path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18. | default | default, bf16, fp8 |
--enable-aiter-allreduce-fusion | Enable aiter allreduce fusion with Residual RMSNorm. | False | bool flag (set to enable) |
--deepep-mode | Select the mode when enable DeepEP MoE, could be normal, low_latency or auto. Default is auto, which means low_latency for decode batch and normal for prefill batch. | auto | normal, low_latency, auto |
--deepep-dispatcher-output-dtype | Select DeepEP dispatcher output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 Series NPUs), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input_global_scale check → dispatcher_output_dtype from quant_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). | auto | auto, bf16, fp8, int8, nvfp4 |
—flashinfer-a2a-dispatch-type | Select FlashInfer A2A dispatcher activation dtype. When omitted, it falls back to the SGLANG_MOE_NVFP4_DISPATCH environment variable. Explicit auto selects mxfp8 for —quantization mxfp8, nvfp4 for modelopt FP4 or hybrid NVFP4 MoE checkpoints, and bf16 otherwise. The SGLANG_MOE_NVFP4_DISPATCH environment variable cannot be set with this argument. | None | auto, bf16, nvfp4, mxfp8 |
--ep-num-redundant-experts | Allocate this number of redundant experts in expert parallel. | 0 | Type: int |
--ep-dispatch-algorithm | The algorithm to choose ranks for redundant experts in expert parallel. | None | Type: str |
--init-expert-location | Initial location of EP experts. | trivial | Type: str |
--enable-eplb | Enable EPLB algorithm | False | bool flag (set to enable) |
--eplb-algorithm | Chosen EPLB algorithm | auto | Type: str |
--eplb-rebalance-num-iterations | Number of iterations to automatically trigger a EPLB re-balance. | 1000 | Type: int |
--eplb-rebalance-layers-per-chunk | Number of layers to rebalance per forward pass. | None | Type: int |
--eplb-min-rebalancing-utilization-threshold | Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0]. | 1.0 | Type: float |
--expert-distribution-recorder-mode | Mode of expert distribution recorder. | None | Type: str |
--expert-distribution-recorder-buffer-size | Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. | None | Type: int |
--expert-balancedness-report-mode | Where to report expert balancedness. | off | off, server_log, prometheus, both |
--deepep-config | Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. | None | Type: str |
--moe-ascend-ops-config | JSON configuration for the experimental ascend_ops MoE backend on the main or speculative draft model. prefill accepts allgather or alltoallv (default alltoallv); decode accepts low_latency or fused_op (default low_latency). Requires both the main and speculative backends to be ascend_ops when a speculative draft model is enabled; the backend is supported on an Ascend NPU. Example: --moe-a2a-backend ascend_ops --speculative-moe-a2a-backend ascend_ops --moe-ascend-ops-config '{"prefill":"allgather","decode":"low_latency"}'. | None | Type: JSON str |
--moe-dense-tp-size | TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. | none | Type: int |
--elastic-ep-backend | Specify the collective communication backend for elastic EP. Currently supports ‘mooncake’. | None | none, mooncake, nixl |
--enable-elastic-expert-backup | Enable elastic EP backend to backup expert weights in DRAM feature. Currently supports ‘mooncake’. | False | bool flag (set to enable) |
--mooncake-ib-device | The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., —mooncake-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled. | None | Type: str |
--enable-waterfill | Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports --deepep-mode auto, normal, or low_latency when used with DeepEP. Use auto or low_latency for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set SGLANG_DISABLE_STATIC_WATERFILL=1 to force dynamic Waterfill with runtime EP all-reduce. | False | bool flag (set to enable) |
--flashinfer-allreduce-fusion-backend | Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. ‘auto’: choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. ‘trtllm’: available on single-node systems only. ‘mnnvl’: available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. | None | auto, trtllm, mnnvl |
--enforce-disable-flashinfer-allreduce-fusion | Enforce disable FlashInfer allreduce fusion. | False | bool flag (set to enable) |
Mamba Cache
| Argument | Description | Defaults | Options |
|---|---|---|---|
--max-mamba-cache-size | The maximum size of the mamba cache. | None | Type: int |
--mamba-ssm-dtype | The data type of the SSM states in mamba cache. If not set, read from the model config. | Auto (from model config) | float32, bfloat16, float16 |
--enable-mamba-cache-stochastic-rounding | Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires —mamba-ssm-dtype float16 and CUDA. With —mamba-backend triton, requires SM100. | False | Type: bool |
--mamba-cache-philox-rounds | Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0. | 0 | Type: int |
--mamba-full-memory-ratio | The ratio of mamba state memory to full kv cache memory. | 0.9 | Type: float |
--mamba-radix-cache-strategy | The strategy to use for mamba scheduler. auto currently defaults to no_buffer. 1. no_buffer does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. extra_buffer supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes 2x for non-spec; 1+(1/(2+speculative_num_draft_tokens))x for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. extra_buffer is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra_buffer supports branching. 3. extra_buffer_lazy lowers extra_buffer’s slot cost by allocating one track slot per request instead of two; the second slot is allocated on demand at track-interval boundaries (for speculative decoding it is reserved ahead of each verify window and committed only for accepted boundary crossings). Compatible with speculative decoding (EAGLE/NGRAM/DSPARK/DFLASH); not supported under PD disaggregation. | auto | auto, no_buffer, extra_buffer, extra_buffer_lazy |
--mamba-track-interval | The interval (in tokens) to track the mamba state during decode. Only used when —mamba-radix-cache-strategy is extra_buffer. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. | 256 | Type: int |
--hybrid-cache-checkpoints-level | Where a hybrid model’s prefix cache checkpoints recurrent (Mamba) state. 2: branching points and request ends only, with a soft cap of one state per path (branching and locked nodes are preserved); suits multi-turn and agentic traffic, where each turn extends the previous one. Host and storage tiers receive published states. For ordinary requests, deferred KV joins the tree at a reusable checkpoint when the request finishes. Streaming sessions retain deferred KV privately and currently do not publish it at turn end. Models that also have sliding-window layers still checkpoint every chunk. 3: every prefill chunk and decode tracking boundary, so a later request can resume from any chunk of an earlier one. Replaces the deprecated —mamba-max-states-per-path, which still applies when set. | 2 | 2, 3 |
--enable-int8-mamba-checkpoint | Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for ~2x cached-prefix capacity at fixed memory. | False | bool flag (set to enable) |
--int8-mamba-ckpt-size | Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size). | None | Type: int |
--mamba-backend | Choose the kernel backend for Mamba SSM operations. Default is ‘triton’. Options: ‘triton’ (default), ‘flashinfer’ (requires FlashInfer with Mamba support). | triton | triton, flashinfer |
--linear-attn-backend | The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by —linear-attn-decode-backend and —linear-attn-prefill-backend. | triton | triton, cutedsl, flashinfer |
--linear-attn-decode-backend | Override the kernel backend for linear attention decode. If not set, uses —linear-attn-backend. | None | triton, cutedsl, flashinfer |
--linear-attn-prefill-backend | Override the kernel backend for linear attention prefill/extend. If not set, uses —linear-attn-backend. | None | triton, cutedsl, flashinfer |
Hierarchical cache
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-hierarchical-cache--enable-hicache | Enable hierarchical cache | False | bool flag (set to enable) |
--enable-unified-cache-external-linker | Link UnifiedRadixCache directly to an external KV store, with no host cache tier. | False | bool flag (set to enable) |
--unified-cache-external-linker-backend | Select the external linker backend. Use dynamic for an installed out-of-tree linker and specify its class and module in —unified-cache-external-linker-config. | mooncake | mooncake, mori, dynamic |
--unified-cache-external-linker-config | JSON object configuring the selected backend. Common options are load_mode (layer_wise, default, or prefetch) and prefetch_policy (best_effort, wait_complete, or timeout, default). Prefetch supports Full/SWA/Mamba within the backend’s existing layout limits and requires PP=1. For dynamic, provide linker (class name), linker_module_path (Python module), and optional linker_extra_config (object). Built-in backends accept storage options directly. You can supply config while the linker is disabled. | None | Type: JSON object |
--hicache-ratio | The ratio of the size of host KV cache memory pool to the size of device pool. | 2.0 | Type: float |
--hicache-size | The size of host KV cache memory pool in gigabytes, which will override the hicache_ratio if set. | 0 | Type: int |
--hicache-write-policy | The write policy of hierarchical cache. | write_through | write_back, write_through, write_through_selective |
--hicache-io-backend | The IO backend for KV cache transfer between CPU and GPU | kernel | direct, kernel, kernel_ascend |
--hicache-mem-layout | The layout of host memory pool for hierarchical cache. | page_first | layer_first, page_first, page_first_direct, page_first_kv_split, page_head, page_unified |
--hicache-storage-backend | The storage backend for hierarchical KV cache. Built-in backends: file, fast_file, mooncake, npu_memcache, hf3fs, nixl, aibrix, tensorcast, seaweedfs. For dynamic backend, use —hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name). | None | file, fast_file, mooncake, hf3fs, nixl, aibrix, dynamic, eic, simm |
--hicache-storage-key-scheme | How L3 storage object keys encode what an object holds. ‘rank-suffix’ (legacy): keys carry the writer’s tp/pp/cp rank, so only a bit-identical topology can reuse the cache. ‘unified’: keys carry topology-free grid coordinates (a namespace digest plus absolute layer and head-group indices), so any topology whose shards tile the same grid shares one cache. Implies —hicache-mem-layout page_unified and —hicache-io-backend kernel. Supported by the file and mooncake backends for a single MHA/MLA pool. To reuse L3 across different TP and PP splits, set the same —hicache-storage-head-partition and —hicache-storage-layer-partition on all instances so each local shard tiles the shared chunk grid. Both backends store each chunk separately and reassemble the reader’s local pages; a page is a cache hit only when all of its chunks are present. Composite caches (including GLM-5.2 indexer and DeepSeek-V4 pools), speculative decoding, and decode context parallelism (DCP) are not supported. | rank-suffix | rank-suffix, unified |
--hicache-storage-head-partition | KV heads per L3 chunk under —hicache-storage-key-scheme unified. A FLEET-WIDE agreement, not a local tuning knob: it fixes chunk boundaries, and the boundaries are part of the key namespace. Set it to total_kv_heads divided by the lcm of the attention-TP sizes in the fleet, so every member’s local head count is a whole number of chunks. Unset, each deployment keys to its own head count and shares only with its own TP size. MLA-family models replicate KV across ranks and ignore it. | None | Type: int |
--hicache-storage-layer-partition | Layers per L3 chunk under —hicache-storage-key-scheme unified; the pipeline-parallel counterpart of —hicache-storage-head-partition and equally fleet-wide. Every stage must START on a multiple of it; only the final stage may end short, and that remainder forms the last chunk. Unset, each stage stores its own layer range and shares only with an identical split. | None | Type: int |
--hicache-storage-prefetch-policy | Control when prefetching from the storage backend should stop. | timeout | best_effort, wait_complete, timeout |
--hicache-storage-backend-extra-config | A dictionary in JSON string format, or a string starting with a @ followed by a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend. | None | Type: str |
Hierarchical sparse attention
| Argument | Description | Defaults | Options |
|---|---|---|---|
--hisparse-config--hierarchical-sparse-attention-extra-config | A dictionary in JSON string format for hierarchical sparse attention configuration. Required fields: algorithm (str), backend (str). All other fields are algorithm-specific and passed to the algorithm constructor. | None | Type: str |
--enable-hisparse | Enable hierarchical sparse attention | False | bool flag (set to enable) |
LMCache
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-lmcache | Using LMCache as an alternative hierarchical cache solution | False | bool flag (set to enable) |
--lmcache-config-file | Path to the LMCache YAML configuration file | None | Type: str |
Paged experts
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-paged-experts | Keep only --paged-experts-num-resident (K) routed experts per MoE layer on the GPU and page the rest from pinned host memory on demand. Single GPU; the experts must run on the triton or Marlin MoE runner (supported quantizations: see the Paged Experts docs). Decode CUDA graphs (full backend) are captured up to K // top_k requests; prefill graphs must be breakable or disabled. | False | bool flag (set to enable) |
--paged-experts-num-resident | Number of routed experts per MoE layer kept resident on the GPU, at least the router’s top-k; or auto (the default): as many as fit in --mem-fraction-static next to a KV cache of --max-total-tokens, else of up to --max-running-requests, if you set it (else 1), requests of --context-length (default: the model’s), at least one. | auto | Type: int or auto |
--paged-experts-enable-disk | Let Paged Experts write to disk under $SGLANG_CACHE_DIR/paged_experts: each MoE layer’s experts are kept there (about the size of the experts), so that later starts read them instead of loading and repacking them, and experts that do not fit the host budget (--paged-experts-host-memory-gb) are served from there. Requires --enable-paged-experts. | False | bool flag (set to enable) |
--paged-experts-host-memory-gb | Pinned host memory (GiB) for the experts of all MoE layers with --enable-paged-experts. When not every expert fits, each layer keeps only some experts in pinned memory and reads the others from its file under $SGLANG_CACHE_DIR/paged_experts, which needs --paged-experts-enable-disk. Unset: every expert is pinned; with --paged-experts-enable-disk, the experts that do not fit what is available at startup within the container’s memory limit go to disk, keeping 10 GiB free. | None | Type: float |
Ktransformers
| Argument | Description | Defaults | Options |
|---|---|---|---|
--kt-weight-path | [ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder. | None | Type: str |
--kt-method | [ktransformers parameter] Quantization formats for CPU execution. | AMXINT4 | Type: str |
--kt-cpuinfer | [ktransformers parameter] The number of CPUInfer threads. | None | Type: int |
--kt-threadpool-count | [ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA). | 2 | Type: int |
--kt-num-gpu-experts | [ktransformers parameter] The number of GPU experts. | None | Type: int |
--kt-max-deferred-experts-per-token | [ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0. | None | Type: int |
Diffusion LLM
| Argument | Description | Defaults | Options |
|---|---|---|---|
--dllm-algorithm | The diffusion LLM algorithm, such as LowConfidence. | None | Type: str |
--dllm-algorithm-config | The diffusion LLM algorithm configurations. Must be a YAML file. | None | Type: str |
--dllm-fdfo | First-Done-First-Out (FDFO) scheduling lets completed requests leave the batch immediately instead of waiting for slower requests, eliminating head-of-line blocking. Enabled by default; pass --no-dllm-fdfo to fall back to synchronous lockstep scheduling. Works with any dLLM algorithm. | True | Type: bool |
Offloading
| Argument | Description | Defaults | Options |
|---|---|---|---|
--cpu-offload-gb | How many GBs of RAM to reserve for CPU offloading. | 0 | Type: int |
--offload-group-size | Number of layers per group in offloading. | -1 | Type: int |
--ple-offload-embedding | Keep the Qwen4-Exp PLE n-gram embedding table in host memory instead of device memory; the gather kernel reads it from a host pointer. Enabled by default for BF16 Qwen4-Exp on CUDA; --no-ple-offload-embedding disables it. | None | Type: bool |
--ple-offload-backend | Host storage for the offloaded PLE table: pinned (CPU pinned memory) or file (a sparse file-backed mmap under --ple-offload-dir, read directly by the gather kernel). Use file on unified-memory devices such as GB10 / DGX Spark, where pinned host memory comes out of the same pool as the weights; it requires a device that reports cudaDevAttrPageableMemoryAccess (GB10, or a discrete GPU with HMM). | pinned | Type: str |
--ple-offload-dir | Directory for the file-backed PLE table when --ple-offload-backend is file. The file is sparse and rewritten on every startup; keep it on fast local storage (NVMe). Rewriting an existing table can be much slower than filling a fresh sparse file. Before restarting, stop all servers using that directory and remove its ple_table_*.bin files to avoid this startup cost. | $SGLANG_CACHE_DIR/ple/<model path> | Type: str |
--offload-num-in-group | Number of layers to be offloaded within a group. | 1 | Type: int |
--offload-prefetch-step | Steps to prefetch in offloading. | 1 | Type: int |
--offload-mode | Mode of offloading. | cpu | Type: str |
--engram-config | Where the DeepSeek-V4.1 Engram tables live, as a JSON object; omitted keys keep their defaults. placement: gpu (default) or cpu (pinned host memory, leaving the HBM to the KV cache). replication: none (default; one table split across the TP group) or per_node (each node holds the whole table; needs cpu and one node). lookup, with replication none: auto (default; SGLang picks the collective), all_reduce, reduce_scatter, all_to_all, or peer_read (read rows from the owning GPU’s shard; needs gpu and CUDA). See Engram tables. | None | Type: JSON object |
Args for multi-item scoring
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-mis | Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires —attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill. | False | bool flag (set to enable) |
Optimization/debug options
| Argument | Description | Defaults | Options |
|---|---|---|---|
--disable-radix-cache | Disable RadixAttention for prefix caching. | False | bool flag (set to enable) |
--cuda-graph-config | Canonical per-phase CUDA graph settings as JSON, e.g. . JSON wins over the per-phase —cuda-graph-* convenience flags and over the legacy flags. Allowed backends: full, breakable, disabled. | None | Type: JSON (dict-of-dicts) |
--cuda-graph-backend-decode | Backend for the decode phase. Folds into cuda_graph_config[decode].backend. | None | full, breakable, disabled |
--cuda-graph-backend-prefill | Backend for the prefill phase. Folds into cuda_graph_config[prefill].backend. | None | breakable, disabled |
--cuda-graph-max-bs-decode | Maximum batch size captured for the decode CUDA graph. | None | Type: int |
--cuda-graph-max-bs-prefill | Maximum batch size captured for the prefill CUDA graph. | None | Type: int |
--cuda-graph-max-seq-len-prefill | Longest sequence a prefill CUDA graph replay admits; longer batches run eager prefill. Folds into cuda_graph_config[prefill].max_seq_len. | None | Type: int |
--cuda-graph-bs-decode | Explicit list of batch sizes to capture for the decode CUDA graph. | None | List[int] |
--cuda-graph-bs-prefill | Explicit list of batch sizes to capture for the prefill CUDA graph. | None | List[int] |
--disable-cuda-graph-padding | Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. | False | bool flag (set to enable) |
--enable-profile-cuda-graph | Enable profiling of cuda graph capture. | False | bool flag (set to enable) |
--cuda-graph-persistence | Persist the CUDA graphs across restarts with Foundry (optional dependency sglang[foundry], package foundry-core): save captures and writes an archive, load rebuilds the graphs from it instead of capturing. See CUDA Graph Persistence. | None | save, load |
--cuda-graph-persistence-config | Foundry TOML for --cuda-graph-persistence (archive directory, address region). Its mode key is ignored: the flag’s mode wins. | None | Type: str |
--debug-cuda-graph | Eager-mode CUDA graph via the breakable backend: graph breaks let every op run eagerly while still going through the capture/replay path. Useful for debugging capture/replay issues. | False | bool flag (set to enable) |
--enable-cudagraph-gc | Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen and paused during capture, which speeds it up and keeps a collection from invalidating the graph. | False | bool flag (set to enable) |
--enable-layerwise-nvtx-marker | Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. | False | bool flag (set to enable) |
--enable-nccl-nvls | Enable NCCL NVLS for prefill heavy requests when available. | False | bool flag (set to enable) |
--enable-symm-mem | Enable NCCL symmetric memory for fast collectives. | False | bool flag (set to enable) |
--disable-flashinfer-cutlass-moe-fp4-allgather | Disables quantize before all-gather for flashinfer cutlass moe. | False | bool flag (set to enable) |
--enable-tokenizer-batch-encode | Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds. | False | bool flag (set to enable) |
--disable-tokenizer-batch-decode | Disable batch decoding when decoding multiple completions. | False | bool flag (set to enable) |
—boundary-reduction | Select the optional FFN output reduction strategy. Unsupported reduce-scatter paths fall back to all-reduce followed by token redistribution. See FFN boundary reduction. | auto | auto, ar, rs, rsv, rs+rsv |
--disable-custom-all-reduce | Disable the custom all-reduce kernel and fall back to NCCL. | False | bool flag (set to enable) |
--enable-mscclpp | Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. | False | bool flag (set to enable) |
--enable-torch-symm-mem | Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. | False | bool flag (set to enable) |
--disable-overlap-schedule | Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. | False | bool flag (set to enable) |
--enable-mixed-chunk | Enabling mixing prefill and decode in a batch when using chunked prefill. | False | bool flag (set to enable) |
--enable-dp-lm-head | Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. | False | bool flag (set to enable) |
--kda-dwdp-size | DWDP for Kimi-K3 KDA projection and shared-expert weights on PD prefill workers (—disaggregation-mode prefill) under attention DP: each group of this many attention-DP ranks holding the same attention-TP slice keeps 1/N of every KDA layer’s [q,k,v,g] in-projection and o_proj (and of the shared experts when they would be replicated), and gathers a layer’s full weights over NVLink just before it runs. Must divide the attention-DP size. | None | Type: int |
--enable-two-batch-overlap | Enabling two micro batches to overlap. | False | bool flag (set to enable) |
--enable-single-batch-overlap | Let computation and communication overlap within one micro batch. | False | bool flag (set to enable) |
--tbo-token-distribution-threshold | The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. | 0.48 | Type: float |
--enable-torch-compile | Optimize the model with torch.compile. Experimental feature. | False | bool flag (set to enable) |
--enable-torch-compile-debug-mode | Enable debug mode for torch compile. | False | bool flag (set to enable) |
--torch-compile-max-bs | Set the maximum batch size when using torch compile. | 32 | Type: int |
--disable-prefill-cuda-graph | Disable the prefill-phase CUDA graph. Convenience for —cuda-graph-backend-prefill=disabled. | False | bool flag (set to enable) |
--disable-decode-cuda-graph | Disable the decode-phase CUDA graph. Convenience for —cuda-graph-backend-decode=disabled. | False | bool flag (set to enable) |
--enable-p2p-check | Enable P2P check for GPU access, otherwise the p2p access is allowed by default. | False | bool flag (set to enable) |
--triton-attention-num-kv-splits | The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. | 8 | Type: int |
--triton-attention-split-tile-size | The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. | None | Type: int |
--num-continuous-decode-steps | Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. | 1 | Type: int |
--delete-ckpt-after-loading | Delete the model checkpoint after loading the model. | False | bool flag (set to enable) |
--enable-memory-saver | Allow saving memory using release_memory_occupation and resume_memory_occupation | False | bool flag (set to enable) |
--enable-weights-cpu-backup | Save model weights to CPU memory during release_weights_occupation and resume_weights_occupation | False | bool flag (set to enable) |
--enable-draft-weights-cpu-backup | Save draft model weights to CPU memory during release_weights_occupation and resume_weights_occupation | False | bool flag (set to enable) |
--allow-auto-truncate | Allow automatically truncating requests that exceed the maximum input length instead of returning an error. | False | bool flag (set to enable) |
--enable-custom-logit-processor | Enable users to pass custom logit processors to the server (disabled by default for security) | False | bool flag (set to enable) |
--enable-watermark | Enable text watermarking with the algorithm --watermark-algorithm selects. | False | bool flag (set to enable) |
--watermark-algorithm | The text watermarking algorithm: gumbel (Aaronson-Gumbel) or synthid (SynthID Text). | gumbel | gumbel, synthid |
--watermark-config | Watermark settings as a JSON file path or an inline JSON object with optional default_enabled and enforce_all fields; gumbel reads key, key_b, context_window, mixing_probability, and max_probability; synthid reads keys, ngram_len, num_candidates, context_history_size, and skip_first_ngram_calls. | None | Type: Optional[str] |
--flashinfer-mla-disable-ragged | Not using ragged prefill wrapper when running flashinfer mla | False | bool flag (set to enable) |
--disable-shared-experts-fusion | Disable shared experts fusion optimization for deepseek v3/r1. | False | bool flag (set to enable) |
--disable-chunked-prefix-cache | Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. | False | bool flag (set to enable) |
--image-processor-backend | Image processor backend. auto lets Transformers select the best available backend. | auto | auto, torchvision, pil |
--enable-return-hidden-states | Enable returning hidden states with responses. | False | bool flag (set to enable) |
--return-hidden-states-mode | Set the maximum hidden-state return mode supported by the server. last allows requests with return_hidden_states=False or last; full also allows return_hidden_states=True. | None | last, full |
--enable-return-aux-hidden-states | Capture auxiliary hidden states on the target model without running a speculative draft. --speculative-algorithm then only names the capture convention (EAGLE3, DFLASH or DSPARK) and is reset to None, so no draft is loaded. Requests with return_hidden_states receive the concatenated aux hidden states in place of the last-layer hidden states. Sets --return-hidden-states-mode to full when unset. Requires pp_size == 1. | False | bool flag (set to enable) |
--return-aux-hidden-state-layer-ids | Target-model layer ids for the selected capture convention. Required for DFLASH and DSPARK; EAGLE3 defaults to the model’s EAGLE3 layers. Draft-checkpoint ids can differ: DFLASH uses zero-based layer-output ids for Llama and Qwen3, but layer-input ids for Muse Glimmer. For a Muse DFlash2DraftModel or MuseGlimmerAssistantModel draft, output-layer id 4 becomes capture id 5; this conversion is not automatic. | None | List[int] |
--enable-return-routed-experts | Enable returning routed experts of each layer with responses. With pipeline parallelism each hop also carries the routed experts of the layers run so far, up to num_layers * top-k int32 ids per token. | False | bool flag (set to enable) |
--scheduler-recv-interval | The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. | 1 | Type: int |
--numa-node | Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. | None | List[int] |
--enable-deterministic-inference | Enable deterministic inference mode with batch invariant ops. | False | bool flag (set to enable) |
--rl-on-policy-target | The training system that SGLang needs to match for true on-policy. | None | fsdp |
--enable-attn-tp-input-scattered | Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. | False | bool flag (set to enable) |
--enable-layernorm-sp | Enable sequence parallelism for prefill LayerNorm and residual regions under pure tensor parallelism. Supports Qwen3 dense and Muse Spark. Muse Spark requires TP4/EP4 or TP8/EP8, NVLink/NVSwitch, and —moe-runner-backend flashinfer_trtllm. Use ModelOpt NVFP4 with —moe-a2a-backend flashinfer_nvlink_one_sided, or enable MegaMoE for NVFP4 or BF16. Attention DP and pipeline parallelism are unsupported; speculative decoding is supported only for Muse Spark DFLASH. | False | bool flag (set to enable) |
--layernorm-sp-min-tokens | With —enable-layernorm-sp, run a pass sequence-parallel only above this token count; smaller passes take the replicated path. 0 runs every extend sequence-parallel. Must be a captured prefill CUDA graph bucket, or at or above the largest one. | 0 | Type: int |
--enable-megamoe-for-layernorm-sp | Use MegaMoE for Muse Spark LayerNorm SP prefill, including BF16 excluded layers. Requires —enable-layernorm-sp and DeepGEMM with post-expert normalization (release/v0.2.3). Prefill and decode share TRT-LLM expert weights; decode uses FlashInfer TRT-LLM and all-reduce. FlashInfer A2A is optional. Mixed chunking is unsupported. | False | bool flag (set to enable) |
--enable-prefill-cp | Enable context parallelism for the prefill phase. Select the compute strategy with —cp-strategy. | False | bool flag (set to enable) |
--cp-strategy | Sharding strategy for prefill CP. zigzag assigns each rank one early and one late sequence block; interleave assigns token indices modulo the CP size. | None | zigzag, interleave |
--cp-kv-layout | KV storage layout for prefill CP; omitted keeps replicated KV storage. | None | Registered KV layout names |
--enable-kv-cache-sharding | Shard KV cache pages across GPUs to reduce duplication and increase cache capacity. Only supported on P/D prefill workers. See KV cache sharing for requirements and examples. | False | bool flag (set to enable) |
--enable-fused-qk-norm-rope | Fuse q/k normalization and rope rotary embedding into one kernel. Enabled by default on XPU and disabled elsewhere; --no-enable-fused-qk-norm-rope disables it. | None | Type: bool |
--enable-precise-embedding-interpolation | Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values. | False | bool flag (set to enable) |
--kv-canary | KV cache canary mode. ‘none’ disables the canary (default). ‘log’ prints them while the server keeps running (production-safe). ‘raise’ fails the server on the first detected mismatch (CI lane). | none | none, log, raise |
--kv-canary-real-data | Check the real KV-cache in the canary. ‘none’ (default) disables the feature. ‘partial’ checks the first 16 bytes of each real-KV slot. ‘all’ checks the full real-KV slot. | none | Type: str |
--kv-canary-sweep-interval | Every N forward steps, run a full-pool sweep. | 0 | Type: int |
--pre-warm-nccl | Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL). | False | bool flag (set to enable) |
--enable-dp-attention-local-control-broadcast | With DP-attention, send control messages to every DP group leader and broadcast within attn_tp_group instead of the full tp_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration. | False | bool flag (set to enable) |
--enforce-shared-experts-fusion | Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). On a Quark MXFP4 Qwen3.5/Qwen3.8 checkpoint that excludes its shared expert from quantization, this also quantizes that shared expert to MXFP4 while loading so it can be fused, which runs its weights and its input activations at MXFP4 where the standalone path ran in BF16 - validate accuracy before using it. Mutually exclusive with —disable-shared-experts-fusion. | False | bool flag (set to enable) |
--enable-return-indexer-topk | Enable returning indexer topk indices of layers with indexer with responses. | False | bool flag (set to enable) |
--disable-attn-tp-gather | Disable scheduler-side attn_tp_gather (the upstream SP path that pads num_tokens to attn_tp_size and pre-allocates a gathered buffer). Use for models that manage SP scatter/gather at the model level (e.g., perform their own all_gather/reduce_scatter inside attention) and do not consume the upstream gathered_buffer. Without this, the cuda graph runner pads num_tokens to attn_tp_size, which can cause kernel autotuners to select wrong-sized variants at small batches. | False | bool flag (set to enable) |
--enable-fused-moe-sum-all-reduce | Enable fused moe triton and sum all reduce. | False | bool flag (set to enable) |
--gc-threshold | Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers. | — | Type: int (one or more) |
--gc-freeze-on-idle-secs | After the scheduler has been idle this many seconds, run a full garbage collection and freeze the surviving objects, so later collections do not scan them. Runs once per idle period. A cache flush or PD role switch unfreezes the heap until the next idle check. Any request, including a /health or /health_generate probe, restarts the idle timer. At most every 10 minutes, an idle freeze also unfreezes first, so frozen garbage is freed and logged. Set to 0 to disable. | 0.0 | Type: float |
FFN boundary reduction
Use--boundary-reduction to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to the dense and MoE FFN stages of decoders built with independently constructed stage boundaries (declare_ffn / append_stages; see Layer boundaries). Other models keep their built-in FFN output communication and ignore this option.
auto selects ar for dense Qwen3 language backbones (including Qwen3-VL wrappers), Moss-VL and MiniCPM-V 4.5; rsv for BailingMoELinear/BailingMoeV2_5, LongCat Flash and Step3-VL; and rs+rsv otherwise. Qwen3-MoE, Qwen3-Next and Qwen3.5 are not matched by the dense Qwen3 rule and resolve to rs+rsv.
With auto, a draft loaded from a different path resolves its own model default; a draft on the target’s model path (the MTP default) inherits the target policy. An explicit value applies to both target and draft. The result is recorded in the speculative_boundary_reduction server args field (no_cli=True); it has no separate CLI flag or independent override during resolution.
This option does not disable required attention or MoE collectives, including expert all-to-all, or select the all-reduce fusion backend. For example, ar can still use fused all-reduce plus residual normalization; use --enforce-disable-flashinfer-allreduce-fusion to disable that FlashInfer fusion.
Dynamic batch tokenizer
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-dynamic-batch-tokenizer | Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. | False | bool flag (set to enable) |
--dynamic-batch-tokenizer-batch-size | [Only used if —enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. | 32 | Type: int |
--dynamic-batch-tokenizer-batch-timeout | [Only used if —enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. | 0.002 | Type: float |
Debug tensor dumps
| Argument | Description | Defaults | Options |
|---|---|---|---|
--debug-tensor-dump-output-folder | The output folder for dumping tensors. | None | Type: str |
--debug-tensor-dump-layers | The layer ids to dump. Dump all layers if not specified. | None | Type: JSON list |
--debug-tensor-dump-input-file | The input filename for dumping tensors | None | Type: str |
PD disaggregation
| Argument | Description | Defaults | Options |
|---|---|---|---|
--disaggregation-mode | Only used for PD disaggregation. “prefill” for prefill-only server, and “decode” for decode-only server. If not specified, it is not PD disaggregated | null | null, prefill, decode |
--disaggregation-transfer-backend | The backend for disaggregation transfer. Default is mooncake. | mooncake | mooncake, nixl, ascend, fake, mori, mooncake_tcp |
--disaggregation-bootstrap-port | Bootstrap server port on the prefill server. Default is 8998. | 8998 | Type: int |
--disaggregation-ib-device | The InfiniBand devices for disaggregation transfer, accepts single device (e.g., —disaggregation-ib-device mlx5_0) or multiple comma-separated devices (e.g., —disaggregation-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. | None | Type: str |
--disaggregation-max-top-logprobs-num | Largest top_logprobs_num a PD request can use: the width of the metadata buffer that carries the first output token’s top logprobs from prefill to decode. Set the same value on both roles. Default is 128. | 128 | Type: int |
--disaggregation-decode-allocation-policy | When to reserve per-request decode KV slots. Set the same policy on prefill and decode. See requirements and tradeoffs. | early | early, prefill_complete |
--disaggregation-decode-enable-offload-kvcache | Enable async KV cache offloading on decode server (PD mode). | False | bool flag (set to enable) |
--num-reserved-decode-tokens | Number of decode tokens that will have memory reserved when adding new request to the running batch. | 512 | Type: int |
--disaggregation-decode-polling-interval | The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. | 1 | Type: int |
--disaggregation-decode-enable-radix-cache | Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with —enable-hisparse, —disaggregation-transfer-backend fake, and speculative algorithms other than EAGLE, EAGLE3, DSPARK, and DFLASH. | False | bool flag (set to enable) |
--optimistic-prefill-attempts | Number of optimistic prefill forward passes that skip the bootstrap wait. | 0 | Type: int |
Encode prefill disaggregation
| Argument | Description | Defaults | Options |
|---|---|---|---|
--encoder-only | For MLLM with an encoder, launch an encoder-only server | False | bool flag (set to enable) |
--language-only | For VLM, load weights for the language model only. | False | bool flag (set to enable) |
--language-model-only | Skip the multimodal encoder entirely: its weights are never loaded and the tower is never built, freeing that GPU memory for KV cache. Multimodal requests are rejected. Unlike —language-only this is a standalone mode, not part of encoder/decoder disaggregation. | False | bool flag (set to enable) |
--encoder-transfer-backend | The backend for encoder disaggregation transfer. Default is zmq_to_scheduler. | zmq_to_scheduler | zmq_to_scheduler, zmq_to_tokenizer, mooncake |
--encoder-urls | List of encoder server urls. | [] | Type: JSON list |
--encoder-bootstrap-port | Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here. | 8997 | Type: int |
--encoder-register-urls | One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: —encoder-register-urls http://prefill0:8997 http://prefill1:8997. Used with —encoder-only servers. | [] | Type: str (one or more) |
--enable-adaptive-dispatch-to-encoder | When enabled, adaptively dispatch: multi-image requests go to encoder in language_only epd mode, single-image requests are processed locally. | False | bool flag (set to enable) |
Custom weight loader
| Argument | Description | Defaults | Options |
|---|---|---|---|
--custom-weight-loader | The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func | None | List[str] |
--weight-loader-disable-mmap | Disable mmap while loading weight using safetensors. | False | bool flag (set to enable) |
--weight-loader-prefetch-checkpoints | Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards in a background thread, reducing total network I/O on shared filesystems (NFS/Lustre) from N*checkpoint to 1*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set enable_multithread_load=true in --model-loader-extra-config to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op). | False | bool flag (set to enable) |
--weight-loader-prefetch-num-threads | Number of threads per rank for checkpoint prefetching. | 4 | Type: int |
--remote-instance-weight-loader-seed-instance-ip | The ip of the seed instance for loading weights from remote instance. | None | Type: str |
--remote-instance-weight-loader-seed-instance-service-port | The service port of the seed instance for loading weights from remote instance. | None | Type: int |
--remote-instance-weight-loader-send-weights-group-ports | The communication group ports for loading weights from remote instance. | None | Type: JSON list |
--remote-instance-weight-loader-backend | The backend for loading weights from remote instance. Can be ‘transfer_engine’, ‘nccl’, or ‘modelexpress’. Default is ‘nccl’. | nccl | transfer_engine, nccl, modelexpress |
--remote-instance-weight-loader-start-seed-via-transfer-engine | Start seed server via transfer engine backend for remote instance weight loader. | False | bool flag (set to enable) |
--weight-loader-drop-cache-after-load | Call posix_fadvise(DONTNEED) on each safetensors shard after loading it. | False | bool flag (set to enable) |
--engine-info-bootstrap-port | Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node. | 6789 | Type: int |
--modelexpress-config | JSON config for ModelExpress P2P weight loading. Keys: “url” (optional gRPC host:port override), “transport” (“nixl” or “transfer_engine”). Example: ’{“url”: “localhost:8001”, “transport”: “nixl”}‘ | None | Type: str |
Configuration file support
| Argument | Description | Defaults | Options |
|---|---|---|---|
--config | Read CLI options from a config file. Must be a YAML file with configuration options. | None | Type: str |
For Multi-Modal
| Argument | Description | Defaults | Options |
|---|---|---|---|
--enable-broadcast-mm-inputs-process | Enable broadcast mm-inputs process in scheduler. | False | bool flag (set to enable) |
--mm-process-config | Multimodal preprocessing config, a json config contains keys: image, video, audio, and video_preprocessing_device, the device the token-space processor decodes and preprocesses a video request on (e.g. {“video_preprocessing_device”: “cpu”}). | {} | Type: JSON / Dict |
--enable-token-space-processor | Enable token-space multimodal preprocessing for supported models. Unsupported models keep their existing processor. Not supported with —encoder-only or —language-only. | False | bool flag (set to enable) |
--allowed-media-domains | Restrict client-supplied HTTP(S) media URLs and redirect destinations to these exact hostnames. | Unrestricted | Space-separated hostnames |
--media-url-max-file-size-mb | Maximum streamed size in MiB for one remote media download. Set to 0 to disable the limit. | 64 | Type: int |
--media-url-timeout | Seconds to download one remote media URL, redirects and body included, for every media type. When unset, REQUEST_TIMEOUT applies, else each loader’s own default. | None | Type: float |
--allow-local-media-files | Allow client media to name local files (paths or file:// URLs). Pass —no-allow-local-media-files to reject them. | True | bool flag |
--mm-enable-dp-encoder | Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically. | False | bool flag (set to enable) |
--limit-mm-data-per-request | Limit the number of multimodal inputs per request. e.g. ’{“image”: 1, “video”: 1, “audio”: 1}‘ | None | Type: JSON / Dict |
--enable-mm-global-cache | Enable Mooncake-backed global multimodal embedding cache on encoder servers so repeated images can reuse cached ViT embeddings instead of recomputing them. | False | bool flag (set to enable) |
For checkpoint decryption
| Argument | Description | Defaults | Options |
|---|---|---|---|
--decrypted-config-file | The path of the decrypted config file. | None | Type: str |
--decrypted-draft-config-file | The path of the decrypted draft config file. | None | Type: str |
--enable-prefix-mm-cache | Enable prefix multimodal cache. Currently only supports mm-only. | False | bool flag (set to enable) |
--uocr-doc-split | Unlimited-OCR v2: opt into approximate page-window document processing with KV reuse. Requires a single data-parallel rank. | False | bool flag (set to enable) |
--uocr-doc-page-window | Unlimited-OCR v2: maximum pages a split window transcribes, excluding lookahead pages. Applies to every image mode. Must be at least 1. | 1 | Type: int |
--uocr-max-page-tokens | Unlimited-OCR v2: output-token limit per page. The built-in page guard enforces it without --enable-custom-logit-processor. Also sizes the default context of a split server. 0 disables the length limit. | 0 | Type: int |
Forward hooks
| Argument | Description | Defaults | Options |
|---|---|---|---|
--forward-hooks | JSON-formatted list of forward hook specifications. Each element must include target_modules (list of glob patterns matched against model.named_modules() names) and hook_factory (Python import path to a factory, e.g. my_package.hooks:make_hook). An optional name field is used for logging, and an optional config object is passed as a dict to the factory. | None | Type: JSON list |
For MindStudio-probe(msProbe) dump
| Argument | Description | Defaults | Options |
|---|---|---|---|
--msprobe-dump-config | The path of the JSON configuration file for msProbe. If specified, enables msProbe dump. | None | Type: str |
