This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments.
Why SGLang for RL Lifecycle?
Let’s embrace a guiding principle from early DeepMind’s RL engineering:Be a library, not a framework.This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle:
Fine-Grained Engine Sleep and Wake Up: facilitate maximum-powered rollout and training
Open-To-Use Refit Functionality: diverse methods for co-location or disaggregation
Easy To Postpone Generation: enable partial rollout and dedicated rollout control
Deterministic Inference: achieve deterministic inference to enable zero training-inference mismatch
Load Balancing Router: cache-aware load-balancing for high-throughput rollout
The following sections cover these aspects in detail.
The memory release/resume, weight update and pause/resume endpoints below (/release_memory_occupation, /resume_memory_occupation, /update_weights_from_*, /begin_weight_update, /end_weight_update, /init_weights_update_group, /destroy_weights_update_group, /update_weight_version, /pause_generation, /continue_generation) are admin endpoints. Launch the server with SGLANG_ADMIN_API_KEY=<key> set (or pass --admin-api-key <key>) and send Authorization: Bearer <key> with each call; without an admin key configured, the server answers HTTP 403 (--api-key does not unlock them).
Fine-Grained Engine Sleep and Wake Up
Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step.Under the hood, the RL team uses CUDA-graph-aware weight offload via torch_memory_saver to preserve virtual memory addresses for graph replay. For details, see: Efficient RL Training - Optimizing Memory Usage in verl.
Server flag
Enable memory saver support when launching the server:
Which memory regions to resume. If omitted, all are resumed.
None
Type: list[str], values: kv_cache, weights
Open-To-Use Refit Functionality
After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang’s weight update utilities, see RL System Deep Thinking: Weight Update Mechanisms.How to choose:
From disk is simplest and best for elastic rollout scaling and checkpointing.
From tensor is best for co-located training/rollout when you can pass in-memory tensors.
From distributed is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB).
Update Weights from Disk
When to use:
Save checkpoint to disk and update weights from disk
Dynamic scaling (new rollout instances can load from the same checkpoint)
Why it works well:This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth.Endpoint:POST /update_weights_from_diskRequest body:
Field
Description
Defaults
Options
model_path
The model path with the new weights.
Required
Type: str
load_format
The format to load the weights.
None
Type: str
abort_all_requests
Abort all running requests before update.
False
Type: bool
weight_version
Optional weight version label tracked by the server.
None
Type: str
is_async
Perform weight load asynchronously.
False
Type: bool
torch_empty_cache
Empty torch cache.
False
Type: bool
keep_pause
Keep scheduler paused after update.
False
Type: bool
recapture_cuda_graph
Recapture CUDA graphs after update.
False
Type: bool
token_step
Trainer step id for rollout bookkeeping.
0
Type: int
flush_cache
Flush KV cache after update.
True
Type: bool
Response body:
Field
Description
Defaults
Options
success
Whether the update succeeded.
-
Type: bool
message
Status / error message.
-
Type: str
num_paused_requests
Number of paused requests during update.
0
Type: int
Python Engine API:engine.update_weights_from_disk(model_path, load_format=None)Diffusion engine (SGLang-Diffusion): The diffusion engine exposes the same POST /update_weights_from_disk endpoint with the following behavior:
All-or-nothing with rollback: if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state.
Offload-aware: when layerwise offload (--dit-layerwise-offload) is enabled, the diffusion offload manager replaces GPU parameters with small torch.empty((1,)) placeholders while real weights live in consolidated pinned CPU buffers. A naive param.data.copy_() would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state.
DTensor-aware: parameters distributed via torch.distributed.tensor (tensor parallelism) are updated through distribute_tensor so that each shard is correctly placed on the right device mesh.
Request body:
Field
Description
Defaults
Options
model_path
The model path with the new weights.
Required
Type: str
flush_cache
Flush TeaCache state after update.
True
Type: bool
target_modules
List of module names to update (e.g. [“transformer”]). If omitted, all nn.Module components are updated.
None
Type: list[str]
Response body:
Field
Description
Defaults
Options
success
Whether the update succeeded.
-
Type: bool
message
Status / error message.
-
Type: str
Note: The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently.
Weight Update Session
update_weights_from_tensor and update_weights_from_distributed must run inside a weight-update session. The update calls only write checkpoint-format values; end_weight_update runs every quantized layer’s process_weights_after_loading once, turning them into kernel layout (Marlin repack, MXFP8 shuffles, …), and runs post_load_weights if no weights were loaded through the engine during the session (P2P/RDMA writes). begin_weight_update first restores the few schemes whose kernel layout changes parameter shapes (today compressed-tensors W4A16 MoE on Marlin, and BF16 MoE on flashinfer_trtllm / flashinfer_trtllm_routed) so the checkpoint-format values fit. One session can span many update calls, so the finalize cost is paid once per refit rather than once per bucket. A call outside a session, a second begin_weight_update, or an end_weight_update without one is rejected with success: false (HTTP 400) and leaves the weights untouched.Endpoints:POST /begin_weight_update, POST /end_weight_updatebegin_weight_update accepts selector: "target", "draft", or "all" (default). Under speculative decoding the same selector on the update endpoints picks which model runners receive the weights; end_weight_update finalizes the runners the session opened.
engine.begin_weight_update()for bucket in buckets: engine.update_weights_from_tensor(bucket)engine.end_weight_update()
Update Weights from Tensor
When to use:
Co-located training and rollout, where training can provide tensors directly
Fast in-memory updates
Important constraints:This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts.Endpoint:POST /update_weights_from_tensorRequest body:
Field
Description
Defaults
Options
serialized_named_tensors
Per-TP serialized tensor payloads.
Required
Type: list[str|bytes]
load_format
Optional load format selector.
None
None, direct, flattened_bucket, or a custom loader path string
flush_cache
Flush KV cache after update.
True
Type: bool
abort_all_requests
Abort all running requests before update.
False
Type: bool
weight_version
Optional version label tracked by the server.
None
Type: str
Note: The serialized tensor payloads must be created with MultiprocessingSerializer.serialize(...) and should be base64-safe strings.Python Engine API:engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True), inside begin_weight_update() / end_weight_update().
Update Weights from Distributed Group
When to use:
Disaggregated training and rollout
NCCL or IB-backed weight broadcast from training workers to rollout workers
How it works:Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group.Initialize weight update groupEndpoint:POST /init_weights_update_groupRequest body:
A request can outlive a version change: the RL flow retracts it with pause_generation, refits, and continues it, or POST /update_weight_version relabels while it is still generating. meta_info then reports which tokens came from which weights:
Ranges are half-open [start, end) over output-token indices, prompt excluded. The usual case is a single span.
--enable-prefill-weight-versions adds prefill_weight_versions, the same span list over the prompt tokens, so KV reused from the prefix cache under older weights is visible.
Easy To Postpone Generation
Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult.SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like APRIL, terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work.pause_generation --- update weights --- continue_generation is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks.
Pause Generation
Endpoint:POST /pause_generationRequest body:
Field
Description
Defaults
Options
mode
Pause mode.
abort
abort, retract, in_place
Modes:
abort: Default behavior, identical to abort endpoint with abort_all set. Pending requests from waiting_queue and running_queue will be returned immediately to the caller.
retract: Put engine in “paused” state. Move running requests back to waiting queue. KV cache can be flushed and recomputed later.
in_place: Put engine in “paused” state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent flush_cache call will be unsuccessful.
Continue Generation
Endpoint:POST /continue_generation
Deterministic Inference
In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: True On-Policy and True On-Policy for VLM. For additional context, see the blog post Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles.Server flag:
sgl-router, SGLang’s router, is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic.Key benefits for RL infrastructure:
Async non-blocking efficiency: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers.
Elasticity and fault tolerance: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption.
Training–Inference alignment: Using sgl-router for both training and inference ensures “What You See Is What You Get.” This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment.
Dynamic load balancing and long-tail mitigation: Unlike static partitioning, sgl-router enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths.
{"docsConfig":{"theme":"aspen","$schema":"https://mintlify.com/docs.json","name":"SGLang Documentation","colors":{"primary":"#d55816","light":"#d55816","dark":"#d55816"},"logo":{"light":"https://mintcdn.com/lmsysorg/iZdDMbLWP1BLEIzC/logo/logo.png?fit=max\u0026auto=format\u0026n=iZdDMbLWP1BLEIzC\u0026q=85\u0026s=b27d467947ee2167744cb07b69badf00","dark":"https://mintcdn.com/lmsysorg/iZdDMbLWP1BLEIzC/logo/logo.png?fit=max\u0026auto=format\u0026n=iZdDMbLWP1BLEIzC\u0026q=85\u0026s=b27d467947ee2167744cb07b69badf00"},"favicon":"https://mintcdn.com/lmsysorg/iZdDMbLWP1BLEIzC/favicon.png?fit=max\u0026auto=format\u0026n=iZdDMbLWP1BLEIzC\u0026q=85\u0026s=5524ba9694253eabf4da70fafa1db208","background":{"decoration":"grid","color":{"light":"#fffcfb","dark":"#1d1d1d"}},"footer":{"socials":{"github":"https://github.com/sgl-project/sglang","x":"https://x.com/lmsysorg","linkedin":"https://www.linkedin.com/company/sgl-project/posts?feedView=all","slack":"https://slack.sglang.io/","discord":"https://discord.gg/4ugb2t6YY2"}},"seo":{"metatags":{"google-site-verification":"bX3ofyYQhraIpAYf4DpyZQXZO_G4xLR_RqeBAKnJA7g","canonical":"https://docs.sglang.io"}},"fonts":{"heading":{"family":"Inter","weight":600},"body":{"family":"Inter","weight":400}},"contextual":{"options":["copy","view","chatgpt","claude","perplexity","mcp","cursor","vscode"],"display":"header"},"styling":{"latex":true},"navigation":{"global":{"anchors":[]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"lmsysorg","hasAudioNarration":false,"actualSubdomain":"lmsysorg","customDomains":["docs.sglang.io"],"gitSource":{"type":"github","owner":"sgl-project","repo":"sglang","deployBranch":"main","contentDirectory":"docs","isPrivate":false,"mountPath":"","label":""},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6afc89373f74a047473d53f3","clientVersion":"0.0.4167","searchFilterCounts":[{"language":"default","tag":"Cookbook","count":343},{"language":"default","tag":"Autoregressive Models","count":126},{"language":"default","tag":"User Guide","count":102},{"language":"default","tag":"Hardware","count":68},{"language":"default","tag":"Hardware Platforms","count":68},{"language":"default","tag":"SGLang Diffusion","count":60},{"language":"default","tag":"Ascend NPUs","count":56},{"language":"default","tag":"Advanced Features","count":49},{"language":"default","tag":"Model Deployment","count":37},{"language":"default","tag":"Diffusion Models","count":36}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableAskAiSearchBar":true},"basePath":"","hasChatPermissions":true,"hasWebMcpPermissions":true,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"treatment","enabled":true,"payload":{"type":"string","value":"treatment"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[{"filePath":"custom.css","content":":where(*){scrollbar-width:none;-ms-overflow-style:none}:where(*)::-webkit-scrollbar{width:0;height:0}:where(pre,code,.code-block,.code-group,#request-example,#response-example,.sg-bench-scroll),:where(pre,code,.code-block,.code-group,#request-example,#response-example,.sg-bench-scroll) *{scrollbar-width:auto;-ms-overflow-style:auto}:where(pre,code,.code-block,.code-group,#request-example,#response-example,.sg-bench-scroll)::-webkit-scrollbar,:where(pre,code,.code-block,.code-group,#request-example,#response-example,.sg-bench-scroll) *::-webkit-scrollbar{width:10px;height:10px}table{width:100%;border-collapse:collapse;table-layout:fixed}table thead tr{border-bottom:2px solid #d55816}table thead th{text-align:left;padding:10px 12px;font-weight:700;white-space:nowrap}table thead th:nth-child(odd){background-color:rgba(255,255,255,0.02)}table thead th:nth-child(even){background-color:rgba(255,255,255,0.05)}table tbody tr:nth-child(odd) td{background-color:rgba(255,255,255,0.02)}table tbody tr:nth-child(even) td{background-color:rgba(255,255,255,0.05)}table tbody td{padding:9px 12px}table tbody td:first-child{font-weight:500}html.dark table thead th:nth-child(odd),[data-theme=\"dark\"] table thead th:nth-child(odd){background-color:rgba(255,255,255,0.02)}html.dark table thead th:nth-child(even),[data-theme=\"dark\"] table thead th:nth-child(even){background-color:rgba(255,255,255,0.05)}html.dark table tbody tr:nth-child(odd) td,[data-theme=\"dark\"] table tbody tr:nth-child(odd) td{background-color:rgba(255,255,255,0.02)}html.dark table tbody tr:nth-child(even) td,[data-theme=\"dark\"] table tbody tr:nth-child(even) td{background-color:rgba(255,255,255,0.05)}.sgd-model-catalog{margin:16px 0 24px;border-top:1px solid rgb(226, 232, 240)}.sgd-model-entry{display:grid;grid-template-columns:minmax(138px, 0.34fr) minmax(0, 1fr);column-gap:28px;min-width:0;padding:17px 0;border-bottom:1px solid rgb(226, 232, 240)}.sgd-model-entry-meta{min-width:0}.sgd-model-entry-meta h3{margin:0;color:rgb(15, 23, 42);font-size:0.9rem;font-weight:650;letter-spacing:0;line-height:1.4}.sgd-model-entry-ids{min-width:0}.sgd-model-id{min-width:0;padding:2px 0}.sgd-model-entry .sgd-model-id code{display:block;min-width:0;padding:0;border:0;background:transparent;box-shadow:none;color:rgb(51, 65, 85);font-size:0.78rem;font-weight:500;line-height:1.5;overflow-wrap:anywhere;white-space:normal}.sgd-model-entry-note{margin:7px 0 0;color:rgb(100, 116, 139);font-size:0.76rem;line-height:1.45}.sgd-model-entry-link{display:inline-flex;gap:4px;align-items:center;margin-top:7px;color:rgb(194, 65, 12);font-size:0.74rem;font-weight:600;text-decoration:none}.sgd-model-entry-link:hover{text-decoration:underline}@media (max-width: 620px){.sgd-model-entry{grid-template-columns:minmax(0, 1fr);row-gap:9px;padding:15px 0}.sgd-model-entry-meta{display:flex;align-items:baseline;justify-content:space-between;gap:16px}.sgd-model-entry-link{flex:none;margin-top:0}}.sgd-chip{display:inline-flex;align-items:center;margin:2px 4px 2px 0;padding:2px 8px;border:1px solid rgba(213, 88, 22, 0.22);border-radius:999px;background:rgba(213, 88, 22, 0.08);color:rgb(154, 52, 18);font-size:0.75rem;font-weight:650;line-height:1.45}.sgd-model-tags{display:flex;flex-wrap:wrap;align-items:center;gap:6px;margin:8px 0 30px}.sgd-model-tags .sgd-chip{margin:0;padding:4px 9px;border:0;background:#eef2ff;color:#3730a3;font-size:12px;font-weight:750;line-height:1.4}.sgd-model-tags+h2{margin-top:0}.prose:has(.sgd-model-tags){overflow-x:clip;overflow-y:visible}.sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled),.sg-command-visualizer-choice[aria-disabled=\"false\"],.sg-command-visualizer-tab){transition:transform 160ms ease-out, border-color 160ms ease-out,\n background-color 160ms ease-out, color 160ms ease-out}@media (hover: hover) and (pointer: fine){.sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled),.sg-command-visualizer-choice[aria-disabled=\"false\"],.sg-command-visualizer-tab):hover{transform:translateY(-1px) scale(1.015)}}.sg-command-visualizer:not(.sgd-command-builder) :where(button:not(:disabled),.sg-command-visualizer-choice[aria-disabled=\"false\"],.sg-command-visualizer-tab):active{transform:translateY(0) scale(0.985)}.sg-command-visualizer:not(.sgd-command-builder) :where(button,.sg-command-visualizer-choice,.sg-command-visualizer-tab):focus-visible{outline:2px solid #e2775f;outline-offset:2px}@media (prefers-reduced-motion: reduce){.sg-command-visualizer:not(.sgd-command-builder) :where(button,.sg-command-visualizer-choice,.sg-command-visualizer-tab){transition:none;transform:none!important}}.sgd-command-builder{container:command-builder / inline-size;--sgdb-surface: white;--sgdb-raised: white;--sgdb-ink: oklch(22% 0.02 258);--sgdb-muted: oklch(50% 0.02 258);--sgdb-line: oklch(90% 0.008 258);--sgdb-brand: oklch(57% 0.15 35);--sgdb-serve: oklch(56% 0.13 40);--sgdb-request: oklch(60% 0.12 62);--sgdb-ok: oklch(52% 0.11 153);--sgdb-warn: oklch(59% 0.12 78);--sgdb-error: oklch(53% 0.16 25);--sgdb-well: oklch(45% 0.03 258 / 0.08);--sgdb-well-inset: inset 0 1px 2px oklch(30% 0.03 258 / 0.07);--sgdb-thumb-bg: linear-gradient(180deg, white, oklch(98.5% 0.003 258));--sgdb-thumb-shadow: 0 0 0 0.5px oklch(30% 0.02 258 / 0.1),\n inset 0 1px 0 rgba(255, 255, 255, 0.9),\n 0 1px 1px oklch(30% 0.02 258 / 0.07),\n 0 4px 12px oklch(30% 0.02 258 / 0.1);--sgdb-pick-hover: oklch(45% 0.03 258 / 0.05);--sgdb-code-bg: oklch(97.5% 0.004 258);--sgdb-code-inset: oklch(30% 0.03 258 / 0.06);--sgdb-ease: cubic-bezier(0.32, 0.72, 0, 1);--sgdb-ease-dur: 200ms;--sgdb-spring: cubic-bezier(0.3, 1.35, 0.45, 1);--sgdb-spring-dur: 460ms;max-width:100%;margin:24px 0 40px;overflow:clip;border:1px solid var(--sgdb-line);border-radius:16px;background:var(--sgdb-surface);box-shadow:0 1px 2px oklch(30% 0.03 258 / 0.04),\n 0 12px 32px -12px oklch(30% 0.03 258 / 0.12);color:var(--sgdb-ink)}.sgd-command-builder *,.sgd-command-builder *::before,.sgd-command-builder *::after{box-sizing:border-box}.sgd-command-builder :where(button,input,select){font:inherit}.sgd-builder-scope-tabs{position:relative;display:grid;grid-template-columns:repeat(3, minmax(0, 1fr));margin:12px 12px 0;padding:3px;border-radius:11px;background:var(--sgdb-well);box-shadow:var(--sgdb-well-inset)}.sgd-builder-scope-tabs::before{position:absolute;top:3px;bottom:3px;left:3px;width:calc((100% - 6px) / 3);border-radius:8px;background:var(--sgdb-thumb-bg);box-shadow:var(--sgdb-thumb-shadow);content:\"\";transition:transform 300ms var(--sgdb-ease)}.sgd-builder-scope-tabs:has(>button:nth-child(2)[data-active=\"true\"])::before{transform:translateX(100%)}.sgd-builder-scope-tabs:has(>button:nth-child(3)[data-active=\"true\"])::before{transform:translateX(200%)}.sgd-builder-scope-tabs button{position:relative;z-index:1;min-width:0;min-height:34px;padding:0 4px;border:0;background:transparent;color:var(--sgdb-muted);cursor:pointer;font-size:12px;font-weight:590;white-space:nowrap;transition:color var(--sgdb-ease-dur) var(--sgdb-ease)}.sgd-builder-scope-tabs button[data-active=\"true\"]{color:var(--sgdb-ink)}.sgd-builder-main{display:flex;min-width:0;flex-direction:column}.sgd-builder-controls,.sgd-builder-output-rail{min-width:0}.sgd-builder-controls{container:builder-controls / inline-size;padding:4px 16px 20px}.sgd-builder-docs-tip{display:block;margin:0;padding:12px 16px;border-top:1px solid var(--sgdb-line);color:var(--sgdb-muted);font-size:12px;line-height:1.5}.sgd-builder-docs-tip a{color:var(--sgdb-serve);text-decoration:underline;text-underline-offset:3px}.sgd-builder-scope-panel{display:grid;gap:0}.sgd-builder-recipe{display:flex;min-width:0;padding:11px 0;gap:12px;align-items:center;justify-content:space-between;border-bottom:1px solid var(--sgdb-line)}.sgd-builder-recipe>div:first-child{display:grid;min-width:0;gap:3px}.sgd-builder-recipe>div:first-child span{color:var(--sgdb-muted);font-size:10.5px}.sgd-builder-recipe>div:first-child strong{font-size:12px;font-weight:590;line-height:1.4}.sgd-builder-recipe>div:last-child{display:flex;flex:none;gap:10px;align-items:center}.sgd-builder-recipe>div:last-child>small{color:var(--sgdb-muted);font-size:10px}.sgd-builder-section{min-width:0;padding:13px 0;border-bottom:1px solid var(--sgdb-line)}.sgd-builder-section:last-child{border-bottom:0}.sgd-builder-section-heading{display:flex;gap:8px;align-items:baseline;justify-content:space-between;margin-bottom:8px}.sgd-builder-section-heading>span{font-size:12px;font-weight:590}.sgd-builder-section-heading>small{color:var(--sgdb-muted);font-size:11px;line-height:1.35;text-align:right}.sgd-builder-hardware-grid,.sgd-builder-choice-grid{display:grid;grid-template-columns:repeat(2, minmax(0, 1fr));gap:2px;padding:3px;border-radius:11px;background:var(--sgdb-well);box-shadow:var(--sgdb-well-inset)}.sgd-builder-hardware,.sgd-builder-choice{position:relative;display:grid;min-width:0;min-height:38px;padding:7px 10px;overflow:hidden;align-content:center;border:0;border-radius:8px;background:transparent;color:color-mix(in oklch, var(--sgdb-ink) 72%, transparent);cursor:pointer;font-size:12px;line-height:1.25;text-align:left;transition:background-color var(--sgdb-ease-dur) var(--sgdb-ease),\n color var(--sgdb-ease-dur) var(--sgdb-ease)}.sgd-builder-hardware{display:flex;min-height:34px;flex-flow:row wrap;align-items:baseline;gap:5px}.sgd-builder-hardware strong{font-size:12.5px;font-weight:590;letter-spacing:-0.01em}.sgd-builder-hardware small,.sgd-builder-choice small{margin-top:1px;color:var(--sgdb-muted);font-size:10.5px}.sgd-builder-hardware small{margin-top:0}.sgd-builder-choice-dot{display:none}.sgd-builder-hardware[data-selected=\"true\"],.sgd-builder-choice[data-selected=\"true\"]{background:var(--sgdb-thumb-bg);box-shadow:var(--sgdb-thumb-shadow);color:var(--sgdb-ink)}.sgd-builder-hardware:disabled,.sgd-builder-choice:disabled,.sgd-builder-choice[data-blocked=\"true\"]{cursor:not-allowed;opacity:0.42}.sgd-builder-blocked-note{margin:8px 2px 0;color:var(--sgdb-warn);font-size:11px;line-height:1.45;animation:sgdb-note-in 220ms var(--sgdb-ease)}@keyframes sgdb-note-in{from{opacity:0;transform:translateY(-2px)}to{opacity:1;transform:translateY(0)}}.sgd-builder-choice[data-soft=\"true\"][data-selected=\"true\"]::after{content:\"Not verified yet\";color:var(--sgdb-warn);font-size:10px;line-height:1.3;margin-top:1px}.sgd-builder-resource-grid{display:grid;gap:8px}.sgd-builder-stepper-field{display:flex;min-height:42px;padding:5px 6px 5px 12px;align-items:center;justify-content:space-between;border:0;border-radius:10px;background:var(--sgdb-well);box-shadow:var(--sgdb-well-inset)}.sgd-builder-stepper-field>div:first-child{display:grid;gap:2px}.sgd-builder-stepper-field span{font-size:12px;font-weight:500}.sgd-builder-stepper-field small{color:var(--sgdb-muted);font-size:10px}.sgd-builder-stepper{display:grid;grid-template-columns:28px 40px 28px;align-items:center;border-radius:8px;background:var(--sgdb-thumb-bg);box-shadow:var(--sgdb-thumb-shadow)}.sgd-builder-stepper button,.sgd-builder-request-stepper button{min-width:28px;min-height:30px;padding:0;border:0;border-radius:8px;background:transparent;color:var(--sgdb-ink);cursor:pointer;font-size:14px;transition:background-color var(--sgdb-ease-dur) var(--sgdb-ease)}.sgd-builder-stepper input{width:40px;height:30px;padding:0;border-right:0.5px solid color-mix(in oklch, var(--sgdb-ink) 10%, transparent);border-left:0.5px solid color-mix(in oklch, var(--sgdb-ink) 10%, transparent);border-top:0;border-bottom:0;border-radius:0;appearance:textfield;background:transparent;color:var(--sgdb-ink);font-size:12px;font-variant-numeric:tabular-nums;font-weight:590;line-height:30px;text-align:center}.sgd-builder-stepper input::-webkit-inner-spin-button,.sgd-builder-stepper input::-webkit-outer-spin-button{margin:0;appearance:none}.sgd-builder-stepper button:disabled,.sgd-builder-request-stepper button:disabled{opacity:0.32;cursor:not-allowed}.sgd-builder-resource-summary{margin:8px 0 0;color:var(--sgdb-muted);font-size:11px;line-height:1.4}.sgd-builder-topology-toggle{margin-top:10px}.sgd-builder-text-action{justify-self:start;min-height:32px;padding:0;border:0;background:transparent;color:var(--sgdb-muted);cursor:pointer;font-size:11px;font-weight:500}.sgd-builder-text-action:disabled{cursor:not-allowed;opacity:0.4}.sgd-builder-advanced{margin-top:12px;padding-top:12px;border-top:1px solid var(--sgdb-line)}.sgd-builder-advanced>p{margin:0 0 12px;color:var(--sgdb-muted);font-size:11px;line-height:1.5}.sgd-builder-topology-inputs{display:grid;gap:8px}.sgd-builder-topology-inputs label{display:grid;gap:5px;color:var(--sgdb-muted);font-size:10.5px}.sgd-builder-topology-inputs select{width:100%;min-height:38px;padding:0 8px;border:1px solid var(--sgdb-line);border-radius:6px;background:var(--sgdb-raised);color:var(--sgdb-ink);font-size:12px}.sgd-builder-messages{margin-top:12px;padding:8px 10px;border-radius:6px;background:color-mix(in oklch, var(--sgdb-warn) 9%, transparent);color:color-mix(in oklch, var(--sgdb-warn) 75%, var(--sgdb-ink))}.sgd-builder-messages[data-state=\"error\"]{background:color-mix(in oklch, var(--sgdb-error) 9%, transparent);color:color-mix(in oklch, var(--sgdb-error) 78%, var(--sgdb-ink))}.sgd-builder-messages :is(p,[data-as=\"p\"]){display:block;margin:0;font-size:11px;line-height:1.45}.sgd-builder-messages :is(p,[data-as=\"p\"])+:is(p,[data-as=\"p\"]){margin-top:5px}.sgd-builder-setting-layout,.sgd-builder-setting-list,.sgd-builder-setting-item{min-width:0}.sgd-builder-setting-item{border-bottom:1px solid var(--sgdb-line)}.sgd-builder-setting-item:last-child{border-bottom:0}.sgd-builder-setting-row{display:grid;grid-template-columns:minmax(76px, auto) minmax(0, 1fr) auto auto;width:100%;min-height:44px;padding:8px;gap:8px;align-items:center;border:0;border-bottom:0;border-radius:8px;background:transparent;transition:background-color var(--sgdb-ease-dur) var(--sgdb-ease),\n color var(--sgdb-ease-dur) var(--sgdb-ease);color:var(--sgdb-ink);cursor:pointer;text-align:left}.sgd-builder-setting-row>span:first-child{color:var(--sgdb-muted);font-size:11px}.sgd-builder-setting-row strong{overflow:hidden;font-size:12px;font-weight:500;text-overflow:ellipsis;white-space:nowrap}.sgd-builder-setting-row small{color:var(--sgdb-ok);font-size:10px;font-weight:500}.sgd-builder-setting-row>span:last-child{color:var(--sgdb-muted);font-size:17px}.sgd-builder-setting-row[data-active=\"true\"]{background:color-mix(in oklch, var(--sgdb-serve) 4%, transparent);color:var(--sgdb-serve)}.sgd-builder-context{padding:16px 0 4px}.sgd-builder-context--inline{padding:12px 4px 16px;border-top:1px solid var(--sgdb-line)}.sgd-builder-context--rail{display:none}.sgd-builder-context-heading{display:flex;align-items:start;justify-content:space-between;gap:12px}.sgd-builder-context-heading>div>span{font-size:13px;font-weight:590}.sgd-builder-context-heading :is(p,[data-as=\"p\"]){display:block;margin:4px 0 0;color:var(--sgdb-muted);font-size:11px;font-weight:400;line-height:1.45}.sgd-builder-context-heading>small{flex:none;color:var(--sgdb-muted);font-size:10px}.sgd-builder-context-options{display:grid;grid-template-columns:repeat(2, minmax(0, 1fr));gap:8px;margin-top:12px}.sgd-builder-context-note{margin-top:12px;padding-top:10px;border-top:1px solid var(--sgdb-line)}.sgd-builder-context-note :is(p,[data-as=\"p\"]){display:block;margin:0;color:var(--sgdb-muted);font-size:11px;line-height:1.5}.sgd-builder-context-note a{display:inline-flex;margin-top:8px;min-height:24px;align-items:center;gap:5px;color:var(--sgdb-serve);font-size:11px;font-weight:500;text-decoration:none;transition:color var(--sgdb-ease-dur) var(--sgdb-ease)}.sgd-builder-context-note a+a{margin-left:16px}.sgd-builder-context-note a svg{flex:none;transition:transform var(--sgdb-ease-dur) var(--sgdb-ease)}.sgd-builder-context-note a:hover{text-decoration:underline;text-decoration-thickness:1px;text-underline-offset:3px}.sgd-builder-context-note a:hover svg{transform:translateX(1px) scale(1.08)}.sgd-builder-scope-panel[data-scope=\"request\"] .sgd-builder-context-note a{color:var(--sgdb-request)}.sgd-builder-request-stepper{display:grid;grid-template-columns:40px 52px 40px auto;gap:6px;align-items:center;margin-top:14px}.sgd-builder-request-stepper input{width:52px;min-width:0;height:34px;padding:0 4px;border:1px solid var(--sgdb-line);border-radius:6px;appearance:textfield;background:var(--sgdb-surface);color:var(--sgdb-ink);font-size:16px;font-variant-numeric:tabular-nums;font-weight:590;text-align:center}.sgd-builder-request-stepper input::-webkit-inner-spin-button,.sgd-builder-request-stepper input::-webkit-outer-spin-button{margin:0;appearance:none}.sgd-builder-request-stepper>span{margin-left:4px;color:var(--sgdb-muted);font-size:11px}.sgd-builder-request-setting{border-bottom:1px solid var(--sgdb-line)}.sgd-builder-request-setting:last-child{border-bottom:0}.sgd-builder-request-setting .sgd-builder-context{padding:16px 0}.sgd-builder-output-rail{display:grid;gap:12px;padding:0 12px 16px;align-items:stretch}.sgd-builder-output{--output-accent: var(--sgdb-serve);display:flex;min-width:0;flex-direction:column;overflow:hidden;border:1px solid var(--sgdb-line);border-radius:12px;background:var(--sgdb-raised);box-shadow:0 1px 2px oklch(30% 0.03 258 / 0.05);transition:border-color 160ms ease-out, transform 160ms ease-out}.sgd-builder-output[data-output=\"request\"]{--output-accent: var(--sgdb-request)}.sgd-builder-output[data-emphasized=\"true\"]{border-color:color-mix(in oklch, var(--output-accent) 24%, var(--sgdb-line))}.sgd-builder-output>header{display:grid;grid-template-columns:24px minmax(0, 1fr) auto;gap:8px;align-items:center;min-height:50px;padding:8px 10px;border-bottom:1px solid var(--sgdb-line)}.sgd-builder-output[data-emphasized=\"true\"]>header{background:linear-gradient(180deg,\n color-mix(in oklch, var(--output-accent) 7%, transparent),\n color-mix(in oklch, var(--output-accent) 2%, transparent))}.sgd-builder-output-index{display:grid;width:20px;height:20px;place-items:center;border:0;border-radius:50%;background:var(--sgdb-well);color:var(--sgdb-muted);font-size:10px;font-weight:590;font-variant-numeric:tabular-nums}.sgd-builder-output-title{display:flex;min-width:0;gap:7px;align-items:baseline}.sgd-builder-output-title strong{font-size:13px;font-weight:590}.sgd-builder-output-title span{color:var(--sgdb-muted);font-size:10.5px}.sgd-builder-status{display:inline-flex;gap:5px;align-items:center;color:var(--sgdb-muted);font-size:10px;white-space:nowrap}.sgd-builder-status>span{width:6px;height:6px;border-radius:50%;background:var(--sgdb-warn);box-shadow:0 0 6px color-mix(in oklch, var(--sgdb-warn) 60%, transparent)}.sgd-builder-status[data-status=\"verified\"]>span{background:var(--sgdb-ok)}.sgd-builder-status[data-status=\"error\"]>span{background:var(--sgdb-error)}.sgd-builder-output-tabs{display:flex;min-height:38px;padding:5px 10px;gap:14px;align-items:center;border-bottom:1px solid var(--sgdb-line)}.sgd-builder-output-tabs button{min-height:28px;padding:0;border:0;border-bottom:1px solid transparent;background:transparent;color:var(--sgdb-muted);cursor:pointer;font-size:10.5px;font-weight:500}.sgd-builder-output-tabs button[data-selected=\"true\"]{border-bottom-color:var(--output-accent);color:var(--sgdb-ink)}.sgd-builder-node-fields{display:grid;grid-template-columns:minmax(0, 1fr) 84px;gap:8px;padding:8px 10px;border-bottom:1px solid var(--sgdb-line)}.sgd-builder-node-fields label{display:grid;min-width:0;gap:4px;color:var(--sgdb-muted);font-size:10px}.sgd-builder-node-fields input{width:100%;min-width:0;min-height:34px;padding:0 7px;border:1px solid var(--sgdb-line);border-radius:6px;background:var(--sgdb-surface);color:var(--sgdb-ink);font-size:11px}.sgd-builder-code{position:relative;display:grid;min-width:0;flex:1 1 auto}.sgd-builder-code>pre{height:100%;max-height:calc(13.95em + 24px);min-width:0;margin:0;padding:12px;overflow:auto;border:0;border-radius:0;background:var(--sgdb-code-bg);box-shadow:inset 0 1px 3px var(--sgdb-code-inset);color:var(--sgdb-ink);font-size:clamp(11.5px, 1.2cqi, 12.5px);line-height:1.55;white-space:pre}.sgd-builder-code>pre::-webkit-scrollbar-corner{background:transparent}.sgd-builder-code>pre.is-expanded{height:auto;max-height:70vh}.sgd-builder-output code{background:transparent;color:inherit;font:inherit;font-variant-ligatures:none}.sgd-builder-output-error{padding:8px 10px;border-top:1px solid color-mix(in oklch, var(--sgdb-error) 35%, var(--sgdb-line));background:color-mix(in oklch, var(--sgdb-error) 8%, transparent);color:var(--sgdb-error);font-size:10.5px;line-height:1.4}.sgd-builder-output>footer{display:flex;min-height:44px;padding:5px 10px;align-items:center;justify-content:space-between;border-top:1px solid var(--sgdb-line)}.sgd-builder-output>footer>div{display:flex;margin-left:auto;gap:12px;align-items:center}.sgd-builder-copy{position:absolute;top:7px;right:7px;display:grid;width:28px;height:28px;padding:0;border:0;border-radius:7px;background:var(--sgdb-thumb-bg);-webkit-backdrop-filter:blur(8px) saturate(1.4);backdrop-filter:blur(8px) saturate(1.4);box-shadow:var(--sgdb-thumb-shadow);color:var(--sgdb-muted);cursor:pointer;opacity:0;place-items:center;transform:translateY(2px) scale(0.9);transition:opacity var(--sgdb-ease-dur) var(--sgdb-ease),\n color var(--sgdb-ease-dur) var(--sgdb-ease),\n transform var(--sgdb-spring-dur) var(--sgdb-spring)}.sgd-builder-code:hover>.sgd-builder-copy,.sgd-builder-copy:focus-visible,.sgd-builder-copy[data-copied=\"true\"]{opacity:1;transform:translateY(0) scale(1)}@media (hover: none){.sgd-builder-copy{opacity:1;transform:none}}.sgd-builder-copy:disabled{cursor:not-allowed;color:color-mix(in oklch, var(--sgdb-muted) 45%, transparent)}.sgd-builder-copy svg{grid-area:1 / 1;pointer-events:none;transition:opacity 140ms var(--sgdb-ease), transform 220ms var(--sgdb-ease)}.sgd-builder-copy .sgd-builder-copy-check{opacity:0;transform:scale(0.5)}.sgd-builder-copy[data-copied=\"true\"]{color:var(--sgdb-ok)}.sgd-builder-copy[data-copied=\"true\"] .sgd-builder-copy-glyph{opacity:0;transform:scale(0.5)}.sgd-builder-copy[data-copied=\"true\"] .sgd-builder-copy-check{opacity:1;transform:scale(1)}@media (hover: hover) and (pointer: fine){.sgd-command-builder button:not(:disabled):not([data-blocked=\"true\"]):hover{color:var(--sgdb-ink)}.sgd-builder-scope-tabs button:not(:disabled):hover{color:var(--sgdb-ink)}.sgd-builder-hardware:not(:disabled):not([data-selected=\"true\"]):hover,.sgd-builder-choice:not(:disabled):not([data-blocked=\"true\"]):not([data-selected=\"true\"]):hover{background:var(--sgdb-pick-hover)}.sgd-builder-setting-row:not(:disabled):hover{background:color-mix(in oklch, var(--sgdb-serve) 5%, transparent)}.sgd-builder-stepper button:not(:disabled):hover,.sgd-builder-request-stepper button:not(:disabled):hover{background:var(--sgdb-pick-hover)}.sgd-builder-copy:not(:disabled):hover{color:var(--sgdb-ink)}.sgd-builder-copy[data-copied=\"true\"]:hover{color:var(--sgdb-ok)}.sgd-builder-text-action:not(:disabled):hover{color:var(--output-accent, var(--sgdb-brand))}}.sgd-command-builder :where(button,input,select,a):focus-visible{outline:2px solid color-mix(in oklch, var(--sgdb-brand) 76%, white);outline-offset:2px}.sgd-command-builder button:not(:disabled):not([data-blocked=\"true\"]):active{transform:scale(0.97);transition-duration:60ms}@container builder-controls (min-width: 560px){.sgd-builder-scope-panel[data-scope=\"base\"]{grid-template-columns:repeat(2, minmax(0, 1fr));column-gap:28px}.sgd-builder-scope-panel[data-scope=\"base\"]>.sgd-builder-recipe{grid-column:1 / -1}.sgd-builder-scope-panel[data-scope=\"base\"] .sgd-builder-resource-grid{grid-template-columns:minmax(0, 1fr)}.sgd-builder-setting-layout{display:grid;grid-template-columns:minmax(240px, 0.9fr) minmax(0, 1.1fr)}.sgd-builder-setting-list{padding-right:16px;border-right:1px solid var(--sgdb-line)}.sgd-builder-context--inline{display:none}.sgd-builder-context--rail{display:block;padding:12px 0 4px 16px}.sgd-builder-request-direct{grid-template-columns:repeat(2, minmax(0, 1fr))}.sgd-builder-request-setting{padding-right:16px;border-right:1px solid var(--sgdb-line);border-bottom:0}.sgd-builder-request-setting+.sgd-builder-request-setting{padding-right:0;padding-left:16px;border-right:0}}@container command-builder (max-width: 717px){.sgd-builder-output-rail{display:contents}.sgd-builder-controls{order:0}.sgd-builder-output{width:calc(100% - 24px);margin:12px 12px 16px;order:1}.sgd-builder-main[data-scope=\"serve\"] .sgd-builder-output[data-output=\"serve\"],.sgd-builder-main[data-scope=\"request\"] .sgd-builder-output[data-output=\"request\"]{order:-1}}@container command-builder (min-width: 718px) and (max-width: 957px){.sgd-builder-output-rail{order:-1;padding-top:12px}.sgd-builder-main[data-scope=\"base\"] .sgd-builder-output-rail{order:1;padding-top:0}.sgd-builder-output[data-emphasized=\"false\"]{order:1}}@container command-builder (min-width: 418px){.sgd-builder-scope-tabs{margin-inline:16px}.sgd-builder-scope-tabs button{font-size:13px}.sgd-builder-hardware-grid{grid-template-columns:repeat(3, minmax(0, 1fr))}.sgd-builder-resource-grid{grid-template-columns:repeat(2, minmax(0, 1fr))}.sgd-builder-topology-inputs{grid-template-columns:repeat(3, minmax(0, 1fr))}}@container command-builder (min-width: 718px){.sgd-builder-scope-panel[data-scope=\"base\"]{grid-template-columns:repeat(12, minmax(0, 1fr));column-gap:16px}.sgd-builder-scope-panel[data-scope=\"base\"]>:where(.sgd-builder-recipe,.sgd-builder-section){grid-column:1 / -1}.sgd-builder-scope-panel[data-scope=\"base\"]>.sgd-builder-section:nth-last-child(-n+2){grid-column:span 6}.sgd-builder-output-rail{grid-template-columns:minmax(0, 1fr)}.sgd-builder-main[data-scope=\"base\"] .sgd-builder-output-rail{grid-template-columns:repeat(2, minmax(0, 1fr))}.sgd-builder-choice-grid[data-density=\"compact\"]{grid-template-columns:repeat(3, minmax(0, 1fr))}}@container command-builder (min-width: 958px){.sgd-builder-scope-tabs{margin-inline:20px}.sgd-builder-scope-tabs button{font-size:14px}.sgd-builder-main{display:grid;grid-template-columns:minmax(0, 38fr) minmax(0, 62fr)}.sgd-builder-controls{padding:8px 20px 24px;border-right:1px solid var(--sgdb-line)}.sgd-builder-output-rail{grid-template-columns:minmax(0, 1fr);padding:16px}.sgd-builder-scope-panel[data-scope=\"base\"]{display:grid;grid-template-columns:minmax(0, 1fr)}.sgd-builder-scope-panel[data-scope=\"base\"]>.sgd-builder-recipe,.sgd-builder-scope-panel[data-scope=\"base\"]>.sgd-builder-section,.sgd-builder-scope-panel[data-scope=\"base\"]>.sgd-builder-section:nth-last-child(-n+2){grid-column:1}.sgd-builder-hardware-grid{grid-template-columns:repeat(2, minmax(0, 1fr))}.sgd-builder-output{box-shadow:0 1px 2px color-mix(in oklch, var(--sgdb-ink) 5%, transparent)}}html.dark .sgd-command-builder,[data-theme=\"dark\"] .sgd-command-builder{--sgdb-surface: oklch(24% 0.015 258);--sgdb-raised: oklch(27% 0.015 258);--sgdb-ink: oklch(94% 0.006 80);--sgdb-muted: oklch(71% 0.012 258);--sgdb-line: rgba(255, 255, 255, 0.09);--sgdb-brand: oklch(70% 0.14 40);--sgdb-serve: oklch(74% 0.11 45);--sgdb-request: oklch(77% 0.1 64);--sgdb-ok: oklch(74% 0.11 153);--sgdb-warn: oklch(76% 0.11 78);--sgdb-error: oklch(73% 0.14 25);--sgdb-well: rgba(0, 0, 0, 0.32);--sgdb-well-inset: inset 0 1px 2px rgba(0, 0, 0, 0.4);--sgdb-thumb-bg: linear-gradient(180deg, rgba(255, 255, 255, 0.14), rgba(255, 255, 255, 0.07));--sgdb-thumb-shadow: 0 0 0 0.5px rgba(255, 255, 255, 0.16),\n inset 0 1px 0 rgba(255, 255, 255, 0.13),\n 0 2px 8px rgba(0, 0, 0, 0.35);--sgdb-pick-hover: rgba(255, 255, 255, 0.06);--sgdb-code-bg: oklch(18% 0.012 258);--sgdb-code-inset: rgba(0, 0, 0, 0.5);box-shadow:inset 0 1px 0 rgba(255, 255, 255, 0.06),\n 0 16px 40px -16px rgba(0, 0, 0, 0.5)}@media (prefers-reduced-motion: reduce){.sgd-command-builder *,.sgd-command-builder *::before,.sgd-command-builder *::after{scroll-behavior:auto!important;transition:none!important;animation:none!important}}html.dark .sgd-model-catalog,[data-theme=\"dark\"] .sgd-model-catalog,html.dark .sgd-model-entry,[data-theme=\"dark\"] .sgd-model-entry{border-color:rgba(148, 163, 184, 0.2)}html.dark .sgd-model-entry-meta h3,[data-theme=\"dark\"] .sgd-model-entry-meta h3{color:rgb(241, 245, 249)}html.dark .sgd-model-entry .sgd-model-id code,[data-theme=\"dark\"] .sgd-model-entry .sgd-model-id code{background:transparent;color:rgb(203, 213, 225)}html.dark .sgd-model-entry-note,[data-theme=\"dark\"] .sgd-model-entry-note{color:rgb(148, 163, 184)}html.dark .sgd-model-entry-link,[data-theme=\"dark\"] .sgd-model-entry-link{color:rgb(251, 146, 60)}html.dark .sgd-chip,[data-theme=\"dark\"] .sgd-chip{border-color:rgba(251, 146, 60, 0.3);background:rgba(251, 146, 60, 0.12);color:rgb(253, 186, 116)}html.dark .sgd-model-tags .sgd-chip,[data-theme=\"dark\"] .sgd-model-tags .sgd-chip{border:0;background:rgba(129, 140, 248, 0.18);color:rgb(199, 210, 254)}.prose strong,.prose b{font-weight:600}:not(pre)>code{background-color:rgba(0, 0, 0, 0.07);font-weight:600}html.dark :not(pre)>code,[data-theme=\"dark\"] :not(pre)>code{background-color:rgba(255, 255, 255, 0.13);font-weight:600}"}],"page":{"pagePath":"docs/advanced_features/sglang_for_rl","slug":"/docs/advanced_features/sglang_for_rl","pageMetadata":{"title":"SGLang for RL Systems","metatags":{"description":"SGLang for RL: engine sleep/wake, weight refit, partial rollout, deterministic inference, cache-aware load balancing for RLHF."},"description":null,"href":"/docs/advanced_features/sglang_for_rl"},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,blockquote:`blockquote`,code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,strong:`strong`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,Heading,Note}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);return _jsxs(_Fragment,{children:[_jsx(_components.p,{children:`This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments.`}),`\n`,_jsx(Heading,{level:`2`,id:`why-sglang-for-rl-lifecycle`,children:`Why SGLang for RL Lifecycle?`}),`\n`,_jsx(_components.p,{children:`Let’s embrace a guiding principle from early DeepMind’s RL engineering:`}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Be a library, not a framework.`})}),`\n`,_jsx(_components.p,{children:`This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Fine-Grained Engine Sleep and Wake Up`}),`: facilitate maximum-powered rollout and training`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Open-To-Use Refit Functionality`}),`: diverse methods for co-location or disaggregation`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Easy To Postpone Generation`}),`: enable partial rollout and dedicated rollout control`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Deterministic Inference`}),`: achieve deterministic inference to enable zero training-inference mismatch`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Load Balancing Router`}),`: cache-aware load-balancing for high-throughput rollout`]}),`\n`]}),`\n`,_jsx(_components.p,{children:`The following sections cover these aspects in detail.`}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`The memory release/resume, weight update and pause/resume endpoints below (`,_jsx(_components.code,{children:`/release_memory_occupation`}),`, `,_jsx(_components.code,{children:`/resume_memory_occupation`}),`, `,_jsx(_components.code,{children:`/update_weights_from_*`}),`, `,_jsx(_components.code,{children:`/begin_weight_update`}),`, `,_jsx(_components.code,{children:`/end_weight_update`}),`, `,_jsx(_components.code,{children:`/init_weights_update_group`}),`, `,_jsx(_components.code,{children:`/destroy_weights_update_group`}),`, `,_jsx(_components.code,{children:`/update_weight_version`}),`, `,_jsx(_components.code,{children:`/pause_generation`}),`, `,_jsx(_components.code,{children:`/continue_generation`}),`) are admin endpoints. Launch the server with `,_jsx(_components.code,{children:`SGLANG_ADMIN_API_KEY=\u003ckey>`}),` set (or pass `,_jsx(_components.code,{children:`--admin-api-key \u003ckey>`}),`) and send `,_jsx(_components.code,{children:`Authorization: Bearer \u003ckey>`}),` with each call; without an admin key configured, the server answers HTTP 403 (`,_jsx(_components.code,{children:`--api-key`}),` does not unlock them).`]})}),`\n`,_jsx(Heading,{level:`2`,id:`fine-grained-engine-sleep-and-wake-up`,children:`Fine-Grained Engine Sleep and Wake Up`}),`\n`,_jsx(_components.p,{children:`Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step.`}),`\n`,_jsxs(_components.p,{children:[`Under the hood, the RL team uses CUDA-graph-aware weight offload via `,_jsx(_components.a,{href:`https://github.com/fzyzcjy/torch_memory_saver`,children:`torch_memory_saver`}),` to preserve virtual memory addresses for graph replay. For details, see: `,_jsx(_components.a,{href:`https://hebiao064.github.io/rl-memory-management`,children:`Efficient RL Training - Optimizing Memory Usage in verl`}),`.`]}),`\n`,_jsx(Heading,{level:`3`,id:`server-flag`,children:`Server flag`}),`\n`,_jsx(_components.p,{children:`Enable memory saver support when launching the server:`}),`\n`,_jsx(CodeBlock,{filename:`Output`,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`--enable-memory-saver`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(Heading,{level:`3`,id:`release-memory`,children:`Release Memory`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /release_memory_occupation`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsx(`tbody`,{children:_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`tags`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Which memory regions to release. If omitted, all are released.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[`Type: list[str], values: `,_jsx(_components.code,{children:`kv_cache`}),`, `,_jsx(_components.code,{children:`weights`})]})]})})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Behavior notes:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`This call asserts there are no ongoing requests. Ensure the engine is idle before calling it.`}),`\n`,_jsxs(_components.li,{children:[`If `,_jsx(_components.code,{children:`kv_cache`}),` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed.`]}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`resume-memory`,children:`Resume Memory`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /resume_memory_occupation`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsx(`tbody`,{children:_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`tags`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Which memory regions to resume. If omitted, all are resumed.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[`Type: list[str], values: `,_jsx(_components.code,{children:`kv_cache`}),`, `,_jsx(_components.code,{children:`weights`})]})]})})]}),`\n`,_jsx(Heading,{level:`2`,id:`open-to-use-refit-functionality`,children:`Open-To-Use Refit Functionality`}),`\n`,_jsxs(_components.p,{children:[`After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang’s weight update utilities, see `,_jsx(_components.a,{href:`https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md`,children:`RL System Deep Thinking: Weight Update Mechanisms`}),`.`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`How to choose:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`From disk`}),` is simplest and best for elastic rollout scaling and checkpointing.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`From tensor`}),` is best for co-located training/rollout when you can pass in-memory tensors.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`From distributed`}),` is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB).`]}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`update-weights-from-disk`,children:`Update Weights from Disk`}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`When to use:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Save checkpoint to disk and update weights from disk`}),`\n`,_jsx(_components.li,{children:`Dynamic scaling (new rollout instances can load from the same checkpoint)`}),`\n`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Why it works well:`})}),`\n`,_jsx(_components.p,{children:`This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth.`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /update_weights_from_disk`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`model_path`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`The model path with the new weights.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`load_format`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`The format to load the weights.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`abort_all_requests`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Abort all running requests before update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`weight_version`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Optional weight version label tracked by the server.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`is_async`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Perform weight load asynchronously.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`torch_empty_cache`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Empty torch cache.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`keep_pause`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Keep scheduler paused after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`recapture_cuda_graph`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Recapture CUDA graphs after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`token_step`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Trainer step id for rollout bookkeeping.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`0`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: int`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`flush_cache`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Flush KV cache after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`True`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]})]})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Response body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`success`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Whether the update succeeded.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`-`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`message`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Status / error message.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`-`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`num_paused_requests`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Number of paused requests during update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`0`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: int`})]})]})]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Python Engine API:`}),` `,_jsx(_components.code,{children:`engine.update_weights_from_disk(model_path, load_format=None)`})]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Diffusion engine (SGLang-Diffusion):`}),` The diffusion engine exposes the same `,_jsx(_components.code,{children:`POST /update_weights_from_disk`}),` endpoint with the following behavior:`]}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`All-or-nothing with rollback:`}),` if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Offload-aware:`}),` when layerwise offload (`,_jsx(_components.code,{children:`--dit-layerwise-offload`}),`) is enabled, the diffusion offload manager replaces GPU parameters with small `,_jsx(_components.code,{children:`torch.empty((1,))`}),` placeholders while real weights live in consolidated pinned CPU buffers. A naive `,_jsx(_components.code,{children:`param.data.copy_()`}),` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`DTensor-aware:`}),` parameters distributed via `,_jsx(_components.code,{children:`torch.distributed.tensor`}),` (tensor parallelism) are updated through `,_jsx(_components.code,{children:`distribute_tensor`}),` so that each shard is correctly placed on the right device mesh.`]}),`\n`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`model_path`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`The model path with the new weights.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`flush_cache`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Flush TeaCache state after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`True`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`target_modules`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[`List of module names to update (e.g. `,_jsx(`code`,{children:`[“transformer”]`}),`). If omitted, all `,_jsx(`code`,{children:`nn.Module`}),` components are updated.`]}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`None`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: list[str]`})]})]})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Response body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`success`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Whether the update succeeded.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`-`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`message`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Status / error message.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`-`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]})]})]}),`\n`,_jsxs(_components.blockquote,{children:[`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Note:`}),` The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently.`]}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`weight-update-session`,children:`Weight Update Session`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`update_weights_from_tensor`}),` and `,_jsx(_components.code,{children:`update_weights_from_distributed`}),` must run inside a weight-update session. The update calls only write checkpoint-format values; `,_jsx(_components.code,{children:`end_weight_update`}),` runs every quantized layer’s `,_jsx(_components.code,{children:`process_weights_after_loading`}),` once, turning them into kernel layout (Marlin repack, MXFP8 shuffles, …), and runs `,_jsx(_components.code,{children:`post_load_weights`}),` if no weights were loaded through the engine during the session (P2P/RDMA writes). `,_jsx(_components.code,{children:`begin_weight_update`}),` first restores the few schemes whose kernel layout changes parameter shapes (today compressed-tensors W4A16 MoE on Marlin, and BF16 MoE on `,_jsx(_components.code,{children:`flashinfer_trtllm`}),` / `,_jsx(_components.code,{children:`flashinfer_trtllm_routed`}),`) so the checkpoint-format values fit. One session can span many update calls, so the finalize cost is paid once per refit rather than once per bucket. A call outside a session, a second `,_jsx(_components.code,{children:`begin_weight_update`}),`, or an `,_jsx(_components.code,{children:`end_weight_update`}),` without one is rejected with `,_jsx(_components.code,{children:`success: false`}),` (HTTP 400) and leaves the weights untouched.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoints:`}),` `,_jsx(_components.code,{children:`POST /begin_weight_update`}),`, `,_jsx(_components.code,{children:`POST /end_weight_update`})]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`begin_weight_update`}),` accepts `,_jsx(_components.code,{children:`selector`}),`: `,_jsx(_components.code,{children:`\"target\"`}),`, `,_jsx(_components.code,{children:`\"draft\"`}),`, or `,_jsx(_components.code,{children:`\"all\"`}),` (default). Under speculative decoding the same selector on the update endpoints picks which model runners receive the weights; `,_jsx(_components.code,{children:`end_weight_update`}),` finalizes the runners the session opened.`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`4`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`4`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`engine.begin_weight_update()`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`for`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` bucket `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`in`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` buckets:`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` engine.update_weights_from_tensor(bucket)`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`engine.end_weight_update()`})}),`\n`]})})}),`\n`,_jsx(Heading,{level:`3`,id:`update-weights-from-tensor`,children:`Update Weights from Tensor`}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`When to use:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Co-located training and rollout, where training can provide tensors directly`}),`\n`,_jsx(_components.li,{children:`Fast in-memory updates`}),`\n`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Important constraints:`})}),`\n`,_jsx(_components.p,{children:`This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts.`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /update_weights_from_tensor`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`serialized_named_tensors`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Per-TP serialized tensor payloads.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: list[str|bytes]`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`load_format`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Optional load format selector.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`None`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[_jsx(`code`,{children:`None`}),`, `,_jsx(`code`,{children:`direct`}),`, `,_jsx(`code`,{children:`flattened_bucket`}),`, or a custom loader path string`]})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`flush_cache`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Flush KV cache after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`True`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`abort_all_requests`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Abort all running requests before update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`weight_version`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Optional version label tracked by the server.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(`code`,{children:`None`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]})]})]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Note:`}),` The serialized tensor payloads must be created with `,_jsx(_components.code,{children:`MultiprocessingSerializer.serialize(...)`}),` and should be base64-safe strings.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Python Engine API:`}),` `,_jsx(_components.code,{children:`engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`}),`, inside `,_jsx(_components.code,{children:`begin_weight_update()`}),` / `,_jsx(_components.code,{children:`end_weight_update()`}),`.`]}),`\n`,_jsx(Heading,{level:`3`,id:`update-weights-from-distributed-group`,children:`Update Weights from Distributed Group`}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`When to use:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`Disaggregated training and rollout`}),`\n`,_jsx(_components.li,{children:`NCCL or IB-backed weight broadcast from training workers to rollout workers`}),`\n`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`How it works:`})}),`\n`,_jsx(_components.p,{children:`Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group.`}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Initialize weight update group`})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /init_weights_update_group`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`master_address`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Group master address.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`master_port`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Group master port.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: int`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`rank_offset`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Offset for local rank mapping.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: int`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`world_size`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Total world size.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: int`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`group_name`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Group name.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`weight_update_group`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`backend`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Communication backend.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`nccl`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]})]})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Update weight`})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /update_weights_from_distributed`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`names`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Parameter names to update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: list[str]`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`dtypes`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Dtype strings for each parameter.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: list[str]`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`shapes`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Tensor shapes.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Required`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: list[list[int]]`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`group_name`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Group name.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`weight_update_group`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`flush_cache`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Flush KV cache after update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`True`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`abort_all_requests`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Abort all running requests before update.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`False`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: bool`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`weight_version`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Optional version label.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`load_format`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Optional format selector.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`None`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[_jsx(_components.code,{children:`None`}),` or `,_jsx(_components.code,{children:`flattened_bucket`})]})]})]})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Destroy weights update group`})}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /destroy_weights_update_group`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsx(`tbody`,{children:_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`group_name`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Group name.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`weight_update_group`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Type: str`})]})})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Python Engine APIs:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:_jsx(_components.code,{children:`engine.init_weights_update_group(...)`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.code,{children:`engine.begin_weight_update(selector=\"all\")`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.code,{children:`engine.update_weights_from_distributed(names, dtypes, shapes, ...)`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.code,{children:`engine.end_weight_update()`})}),`\n`,_jsx(_components.li,{children:_jsx(_components.code,{children:`engine.destroy_weights_update_group(group_name)`})}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`per-token-weight-version-attribution`,children:`Per-Token Weight Version Attribution`}),`\n`,_jsxs(_components.p,{children:[`A request can outlive a version change: the RL flow retracts it with `,_jsx(_components.code,{children:`pause_generation`}),`, refits, and continues it, or `,_jsx(_components.code,{children:`POST /update_weight_version`}),` relabels while it is still generating. `,_jsx(_components.code,{children:`meta_info`}),` then reports which tokens came from which weights:`]}),`\n`,_jsx(CodeBlock,{filename:`Output`,numberOfLines:`5`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`5`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"weight_version\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"42\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"weight_versions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"version\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"41\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"start\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`0`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"end\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`57`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`},`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"version\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"42\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"start\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`57`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"end\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`128`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})}),`\n`]})})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[`Ranges are half-open `,_jsx(_components.code,{children:`[start, end)`}),` over output-token indices, prompt excluded. The usual case is a single span.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`--enable-prefill-weight-versions`}),` adds `,_jsx(_components.code,{children:`prefill_weight_versions`}),`, the same span list over the prompt tokens, so KV reused from the prefix cache under older weights is visible.`]}),`\n`]}),`\n`,_jsx(Heading,{level:`2`,id:`easy-to-postpone-generation`,children:`Easy To Postpone Generation`}),`\n`,_jsx(_components.p,{children:`Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult.`}),`\n`,_jsxs(_components.p,{children:[`SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like `,_jsx(_components.a,{href:`https://arxiv.org/abs/2509.18521`,children:`APRIL`}),`, terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work.`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`pause_generation`}),` --- update weights --- `,_jsx(_components.code,{children:`continue_generation`}),` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks.`]}),`\n`,_jsx(Heading,{level:`3`,id:`pause-generation`,children:`Pause Generation`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /pause_generation`})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Request body:`})}),`\n`,_jsxs(`table`,{style:{width:`100%`,borderCollapse:`collapse`,tableLayout:`fixed`},children:[_jsxs(`colgroup`,{children:[_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}}),_jsx(`col`,{style:{width:`25%`}})]}),_jsx(`thead`,{children:_jsxs(`tr`,{style:{borderBottom:`2px solid #d55816`},children:[_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Field`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Description`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.02)`},children:`Defaults`}),_jsx(`th`,{style:{textAlign:`left`,padding:`10px 12px`,fontWeight:700,whiteSpace:`nowrap`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Options`})]})}),_jsx(`tbody`,{children:_jsxs(`tr`,{children:[_jsx(`td`,{style:{padding:`9px 12px`,fontWeight:500,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`mode`})}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:`Pause mode.`}),_jsx(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.02)`},children:_jsx(_components.code,{children:`abort`})}),_jsxs(`td`,{style:{padding:`9px 12px`,backgroundColor:`rgba(255,255,255,0.05)`},children:[_jsx(_components.code,{children:`abort`}),`, `,_jsx(_components.code,{children:`retract`}),`, `,_jsx(_components.code,{children:`in_place`})]})]})})]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Modes:`})}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`abort`}),`: Default behavior, identical to `,_jsx(_components.code,{children:`abort`}),` endpoint with `,_jsx(_components.code,{children:`abort_all`}),` set. Pending requests from `,_jsx(_components.code,{children:`waiting_queue`}),` and `,_jsx(_components.code,{children:`running_queue`}),` will be returned immediately to the caller.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`retract`}),`: Put engine in “paused” state. Move running requests back to waiting queue. KV cache can be flushed and recomputed later.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`in_place`}),`: Put engine in “paused” state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `,_jsx(_components.code,{children:`flush_cache`}),` call will be unsuccessful.`]}),`\n`]}),`\n`,_jsx(Heading,{level:`3`,id:`continue-generation`,children:`Continue Generation`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Endpoint:`}),` `,_jsx(_components.code,{children:`POST /continue_generation`})]}),`\n`,_jsx(Heading,{level:`2`,id:`deterministic-inference`,children:`Deterministic Inference`}),`\n`,_jsx(_components.p,{children:`In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.`}),`\n`,_jsxs(_components.p,{children:[`SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: `,_jsx(_components.a,{href:`https://github.com/radixark/miles/tree/main/examples/true_on_policy`,children:`True On-Policy`}),` and `,_jsx(_components.a,{href:`https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm`,children:`True On-Policy for VLM`}),`. For additional context, see the blog post `,_jsx(_components.a,{href:`https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md`,children:`Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles`}),`.`]}),`\n`,_jsx(_components.p,{children:_jsx(_components.strong,{children:`Server flag:`})}),`\n`,_jsx(CodeBlock,{filename:`Output`,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`--enable-deterministic-inference`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsxs(_components.p,{children:[`For more details, see `,_jsx(_components.a,{href:`/docs/advanced_features/deterministic_inference`,children:`Deterministic Inference`})]}),`\n`,_jsx(Heading,{level:`2`,id:`load-balancing-router`,children:`Load Balancing Router`}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`sgl-router`}),`, SGLang’s router, is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic.`]}),`\n`,_jsx(_components.p,{children:`Key benefits for RL infrastructure:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Async non-blocking efficiency`}),`: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Elasticity and fault tolerance`}),`: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Training–Inference alignment`}),`: Using `,_jsx(_components.code,{children:`sgl-router`}),` for both training and inference ensures “What You See Is What You Get.” This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Dynamic load balancing and long-tail mitigation`}),`: Unlike static partitioning, `,_jsx(_components.code,{children:`sgl-router`}),` enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths.`]}),`\n`]}),`\n`,_jsxs(_components.p,{children:[`For deployment and configuration, see: `,_jsx(_components.a,{href:`/docs/advanced_features/dp_dpa_router_guide`,children:`DP, DPA and sgl-router`})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"SGLang for RL Systems","metatags":{"description":"SGLang for RL: engine sleep/wake, weight refit, partial rollout, deterministic inference, cache-aware load balancing for RLHF."},"description":null,"href":"/docs/advanced_features/sglang_for_rl"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Why SGLang for RL Lifecycle?","slug":"why-sglang-for-rl-lifecycle","depth":2,"children":[]},{"title":"Fine-Grained Engine Sleep and Wake Up","slug":"fine-grained-engine-sleep-and-wake-up","depth":2,"children":[{"title":"Server flag","slug":"server-flag","depth":3,"children":[]},{"title":"Release Memory","slug":"release-memory","depth":3,"children":[]},{"title":"Resume Memory","slug":"resume-memory","depth":3,"children":[]}]},{"title":"Open-To-Use Refit Functionality","slug":"open-to-use-refit-functionality","depth":2,"children":[{"title":"Update Weights from Disk","slug":"update-weights-from-disk","depth":3,"children":[]},{"title":"Weight Update Session","slug":"weight-update-session","depth":3,"children":[]},{"title":"Update Weights from Tensor","slug":"update-weights-from-tensor","depth":3,"children":[]},{"title":"Update Weights from Distributed Group","slug":"update-weights-from-distributed-group","depth":3,"children":[]},{"title":"Per-Token Weight Version Attribution","slug":"per-token-weight-version-attribution","depth":3,"children":[]}]},{"title":"Easy To Postpone Generation","slug":"easy-to-postpone-generation","depth":2,"children":[{"title":"Pause Generation","slug":"pause-generation","depth":3,"children":[]},{"title":"Continue Generation","slug":"continue-generation","depth":3,"children":[]}]},{"title":"Deterministic Inference","slug":"deterministic-inference","depth":2,"children":[]},{"title":"Load Balancing Router","slug":"load-balancing-router","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"lmsysorg","siteName":"SGLang Documentation","title":"SGLang for RL Systems - SGLang Documentation","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-11-16T07:58:01.527Z","originalFileLocation":"docs/advanced_features/sglang_for_rl.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"application-name\" content=\"SGLang Documentation\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"SGLang for RL Systems - SGLang Documentation\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"SGLang Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://lmsysorg.mintlify.site/_mintlify/api/og/v2?division=Advanced+Features\u0026amp;title=SGLang+for+RL+Systems\u0026amp;theme=182b9b387c023e0456121174\u0026amp;signature=d864c2898382552f70b2a103ef02a218ca71addc0f320ac0749edeae23f32081\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"SGLang for RL Systems - SGLang Documentation\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://lmsysorg.mintlify.site/_mintlify/api/og/v2?division=Advanced+Features\u0026amp;title=SGLang+for+RL+Systems\u0026amp;theme=182b9b387c023e0456121174\u0026amp;signature=d864c2898382552f70b2a103ef02a218ca71addc0f320ac0749edeae23f32081\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/lmsysorg/HxNR9zohmNorO5Pc/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"SGLang Documentation\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#d55816\">\u003cmeta data-mint-head name=\"google-site-verification\" content=\"bX3ofyYQhraIpAYf4DpyZQXZO_G4xLR_RqeBAKnJA7g\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/docs/advanced_features/sglang_for_rl.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.sglang.io/#organization\",\"name\":\"SGLang Documentation\",\"url\":\"https://docs.sglang.io\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/lmsysorg/iZdDMbLWP1BLEIzC/logo/logo.png?fit=max\u0026auto=format\u0026n=iZdDMbLWP1BLEIzC\u0026q=85\u0026s=b27d467947ee2167744cb07b69badf00\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.sglang.io#website\",\"name\":\"SGLang Documentation\",\"url\":\"https://docs.sglang.io\",\"publisher\":{\"@id\":\"https://docs.sglang.io/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl#webpage\",\"url\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl\",\"name\":\"SGLang for RL Systems\",\"dateModified\":\"2026-11-16T07:58:01.527Z\",\"isPartOf\":{\"@id\":\"https://docs.sglang.io#website\"},\"breadcrumb\":{\"@id\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Advanced Features\",\"item\":\"https://docs.sglang.io/docs/advanced_features/overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SGLang for RL Systems\",\"item\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl#article\",\"headline\":\"SGLang for RL Systems\",\"name\":\"SGLang for RL Systems\",\"url\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl\",\"mainEntityOfPage\":{\"@id\":\"https://docs.sglang.io/docs/advanced_features/sglang_for_rl#webpage\"},\"dateModified\":\"2026-11-16T07:58:01.527Z\",\"publisher\":{\"@id\":\"https://docs.sglang.io/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.sglang.io#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"SGLang Documentation\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"SGLang Documentation\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/docs/advanced_features/sglang_for_rl.md","siteName":"SGLang Documentation","nav":{"global":{"anchors":[]},"tabs":[{"tab":"Get Started","pages":[{"title":"Get Started","href":""}]},{"tab":"User Guide","groups":[{"group":"Basic Usage","icon":"book-open","pages":[{"title":"Basic Usage","href":"/docs/basic_usage/overview"},{"title":"Tutorial: Sending a request","href":"/docs/basic_usage/send_request"},{"group":"OpenAI-Compatible APIs","pages":[{"title":"OpenAI-Compatible APIs","href":"/docs/basic_usage/openai_api"},{"title":"OpenAI APIs - Completions","href":"/docs/basic_usage/openai_api_completions"},{"title":"OpenAI APIs - Vision","href":"/docs/basic_usage/openai_api_vision"},{"title":"OpenAI APIs - Embedding","href":"/docs/basic_usage/openai_api_embeddings"}]},{"title":"Anthropic-Compatible API","href":"/docs/basic_usage/anthropic_api"},{"title":"Ollama-Compatible API","href":"/docs/basic_usage/ollama_api"},{"title":"Offline Engine API","href":"/docs/basic_usage/offline_engine_api"},{"title":"SGLang Native APIs","href":"/docs/basic_usage/native_api"},{"title":"Sampling Parameters","href":"/docs/basic_usage/sampling_params"}]},{"group":"Advanced Features","icon":"gears","pages":[{"title":"Advanced Features","href":"/docs/advanced_features/overview"},{"title":"Server Arguments","href":"/docs/advanced_features/server_arguments"},{"title":"Session-Aware Radix Cache","href":"/docs/advanced_features/session_radix_cache"},{"title":"Radix Cache Eviction Policies","href":"/docs/advanced_features/radix_eviction_policy"},{"title":"Hybrid Model Prefix-Cache Checkpoints","href":"/docs/advanced_features/hybrid_cache_checkpoints"},{"title":"Hyperparameter Tuning","href":"/docs/advanced_features/hyperparameter_tuning"},{"title":"Text Watermarking","href":"/docs/advanced_features/text_watermarking"},{"title":"Attention Backend","href":"/docs/advanced_features/attention_backend"},{"title":"HiSparse: Hierarchical Sparse Attention","href":"/docs/advanced_features/hisparse_guide"},{"title":"Speculative Decoding","href":"/docs/advanced_features/speculative_decoding"},{"title":"Adaptive Speculative Decoding","href":"/docs/advanced_features/adaptive_speculative_decoding"},{"title":"Structured Outputs","href":"/docs/advanced_features/structured_outputs"},{"title":"Structured Outputs For Reasoning Models","href":"/docs/advanced_features/structured_outputs_for_reasoning_models"},{"title":"Tool Parser","href":"/docs/advanced_features/tool_parser"},{"title":"Reasoning Parser","href":"/docs/advanced_features/separate_reasoning"},{"title":"Thinking Budget","href":"/docs/advanced_features/thinking_budget"},{"title":"Quantization","href":"/docs/advanced_features/quantization"},{"title":"Quantized KV Cache","href":"/docs/advanced_features/quantized_kv_cache"},{"title":"Reasoning-Aware Compression","href":"/docs/advanced_features/reasoning_aware_compression"},{"title":"DP, DPA and sgl-router","href":"/docs/advanced_features/dp_dpa_router_guide"},{"title":"Prefill Context Parallelism","href":"/docs/advanced_features/prefill_cp"},{"title":"KV Cache Sharing","href":"/docs/advanced_features/kv_cache_sharing"},{"title":"Decode Context Parallelism","href":"/docs/advanced_features/dcp"},{"title":"Expert Parallelism","href":"/docs/advanced_features/expert_parallelism"},{"title":"Paged Experts","href":"/docs/advanced_features/paged_experts"},{"title":"Engram Tables","href":"/docs/advanced_features/engram"},{"title":"LoRA Serving","href":"/docs/advanced_features/lora"},{"title":"PD Disaggregation","href":"/docs/advanced_features/pd_disaggregation"},{"title":"EPD Disaggregation","href":"/docs/advanced_features/epd_disaggregation"},{"title":"Pipeline Parallelism for Long Context","href":"/docs/advanced_features/pipeline_parallelism"},{"group":"Hierarchical KV Caching (HiCache)","pages":[{"title":"Hierarchical KV Caching (HiCache)","href":"/docs/advanced_features/hicache"},{"title":"SGLang HiCache Best Practices","href":"/docs/advanced_features/hicache_best_practices"},{"title":"HiCache System Design and Optimization","href":"/docs/advanced_features/hicache_design"},{"title":"Runtime Attach/Detach HiCache Storage Backend (No Restart)","href":"/docs/advanced_features/hicache_storage_runtime_attach_detach"}]},{"group":"Unified Memory","pages":[{"title":"Unified Memory for Hybrid Models","href":"/docs/advanced_features/unified_memory"},{"title":"Unified Memory Design","href":"/docs/advanced_features/unified_memory_design"}]},{"title":"Hybrid Model Memory Pools","href":"/docs/advanced_features/hybrid_model_memory"},{"title":"SGLang Simulator","href":"/docs/advanced_features/sglang_simulator"},{"title":"Query VLM with Offline Engine","href":"/docs/advanced_features/vlm_query"},{"title":"DP for Multi-Modal Encoder in SGLang","href":"/docs/advanced_features/dp_for_multi_modal_encoder"},{"title":"Cuda Graph for Multi-Modal Encoder in SGLang","href":"/docs/advanced_features/cuda_graph_for_multi_modal_encoder"},{"title":"Breakable CUDA Graph","href":"/docs/advanced_features/breakable_cuda_graph"},{"title":"llm-d","href":"/docs/advanced_features/llm-d"},{"title":"Deterministic Inference","href":"/docs/advanced_features/deterministic_inference"},{"title":"Observability","href":"/docs/advanced_features/observability"},{"title":"Model Loading","href":"/docs/advanced_features/model_loading"},{"title":"Loading Models from Object Storage","href":"/docs/advanced_features/object_storage"},{"title":"Checkpoint Engine Integration","description":null,"href":"/docs/advanced_features/checkpoint_engine"},{"title":"SGLang for RL Systems","href":"/docs/advanced_features/sglang_for_rl"}]},{"group":"Supported Models","icon":"cubes","pages":[{"title":"Supported models","description":"See which families of SGLang-compatible models are actively maintained.","mode":"wide","href":"/docs/supported-models"},{"group":"Text Generation","pages":[{"title":"Large Language Models","href":"/docs/supported-models/generative_models"},{"title":"Multimodal Language Models","href":"/docs/supported-models/multimodal_language_models"},{"title":"Diffusion language models","href":"/docs/supported-models/diffusion_language_models"}]},{"group":"Retrieval and Ranking","pages":[{"title":"Embedding models","href":"/docs/supported-models/embedding_models"},{"title":"Rerank models","href":"/docs/supported-models/rerank_models"},{"title":"Classification Models","href":"/docs/supported-models/classify_models"}]},{"group":"Specialized Models","pages":[{"title":"Reward models","href":"/docs/supported-models/reward_models"},{"title":"Decision models","href":"/docs/supported-models/decision_models"}]},{"group":"Extending SGLang","pages":[{"title":"How to Support New Models","href":"/docs/supported-models/support_new_models"},{"title":"Transformers Fallback in SGLang","href":"/docs/supported-models/transformers_fallback"},{"title":"Use Models From ModelScope","href":"/docs/supported-models/modelscope"},{"title":"MindSpore Models","href":"/docs/supported-models/mindspore_models"}]}]},{"group":"Developer Guide","icon":"code","pages":[{"title":"Developer Guide","href":"/docs/developer_guide/overview"},{"title":"Contribution Guide","mode":"wide","href":"/docs/developer_guide/contribution_guide"},{"title":"Add an out-of-tree serve backend","href":"/docs/developer_guide/serve_backend_plugins"},{"group":"Development","pages":[{"title":"Development Guide Using Docker","sidebarTitle":"Using Docker","href":"/docs/developer_guide/development_guide_using_docker"},{"title":"Development Guide for JIT Kernels","sidebarTitle":"JIT Kernels","href":"/docs/developer_guide/development_jit_kernel_guide"},{"title":"Quantization Contribution Guide","href":"/docs/developer_guide/quantization_contribution_guide"},{"title":"Layer boundaries","href":"/docs/developer_guide/layer_boundary"}]},{"group":"Benchmarking","pages":[{"title":"Benchmark and Profiling","href":"/docs/developer_guide/benchmark_and_profiling"},{"title":"Bench Serving Guide","href":"/docs/developer_guide/bench_serving"}]},{"title":"Evaluating New Models with SGLang","href":"/docs/developer_guide/evaluating_new_models"},{"title":"MSProbe Debugging Guide","href":"/docs/developer_guide/msprobe_debugging_guide"}]},{"group":"References","icon":"bookmark","pages":[{"title":"References","href":"/docs/references/overview"},{"title":"Troubleshooting and Frequently Asked Questions","href":"/docs/references/faq"},{"title":"Environment Variables","href":"/docs/references/environment_variables"},{"title":"Production Metrics","href":"/docs/references/production_metrics"},{"title":"Production Request Tracing","href":"/docs/references/production_request_trace"},{"group":"Deployment","pages":[{"title":"Multi-Node Deployment","href":"/docs/references/multi_node_deployment/multi_node_index"},{"title":"Multi-Node Deployment","href":"/docs/references/multi_node_deployment/multi_node"},{"title":"Deploy On Kubernetes","href":"/docs/references/multi_node_deployment/deploy_on_k8s"},{"title":"LWS Based PD Deploy","href":"/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy"},{"title":"DeepSeekV32-Exp RBG Based PD Deploy","href":"/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd"},{"title":"Amazon SageMaker AI","href":"/docs/basic_usage/aws_sagemaker"}]},{"title":"Custom Chat Template","href":"/docs/references/custom_chat_template"},{"group":"Frontend Language","pages":[{"title":"Frontend Language","href":"/docs/references/frontend/frontend_index"},{"title":"SGLang Frontend Language","href":"/docs/references/frontend/frontend_tutorial"},{"title":"Choices Methods in SGLang","href":"/docs/references/frontend/choices_methods"}]},{"group":"Cookbook","pages":[{"title":"Server Arguments","href":"/cookbook/base/reference/server_arguments"}]},{"title":"Post-Training Integration","href":"/docs/references/post_training_integration"},{"title":null,"href":"/docs/references/nightly_precision_regression"}]}]},{"tab":"Hardware","pages":[{"title":"Hardware","href":"/docs/hardware-platforms/overview"}]},{"tab":"Cookbook","pages":[{"title":"Cookbook","href":"/cookbook/intro"}]},{"tab":"SGLang Diffusion","pages":[{"title":"SGLang Diffusion","href":"/docs/sglang-diffusion"}]}]},"basePath":"","title":"SGLang for RL Systems - SGLang Documentation","buildId":"dpl_9RFrWkXdFTJCytfx6jAofvAaEXr6","deploymentId":"dpl_9RFrWkXdFTJCytfx6jAofvAaEXr6","dynamics":[],"chunks":[]}