Cosmos3 - SGLang Documentation

1. Model Introduction

NVIDIA Cosmos3 is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler. Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository.

Model	Status	Notes
`nvidia/Cosmos3-Nano`	Supported	T2I, T2V, I2V, V2V, joint sound, and action
`nvidia/Cosmos3-Super`	Supported	T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint
`nvidia/Cosmos3-Super-Text2Image`	Supported	T2I-specialized checkpoint
`nvidia/Cosmos3-Super-Image2Video`	Supported	I2V-specialized checkpoint
`nvidia/Cosmos3-Nano-Policy-DROID`	Supported	DROID policy action generation
`nvidia/Cosmos3-Edge`	Supported	4B dense model for T2I, T2V, I2V, V2V, and action generation
`nvidia/Cosmos3-Edge-Policy-DROID`	Supported	4B DROID policy action generation
`nvidia/Cosmos3-Super-Text2Image-4Step`	Supported	64B T2I checkpoint distilled to a fixed 4-step schedule
`nvidia/Cosmos3-Super-Image2Video-4Step`	Supported	64B I2V checkpoint distilled to a fixed 4-step schedule

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native FlowUniPCMultistepScheduler; distilled checkpoints use the fixed sigma schedule stored in the checkpoint. The default flow_shift is 3.0 for T2I, 10.0 for non-Edge video and all action modes, and 3.0 for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level flow_shift.

2. Installation

Install SGLang with the diffusion dependencies:

Command

pip install -e "python[diffusion]"

Cosmos3 guardrails are enabled by default when the package is available:

Command

pip install "cosmos-guardrail==0.3.1"

cosmos-guardrail downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 before starting the server. There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the _pickle.UnpicklingError error occurs during startup, you should change weight_only=True to weights_only=False parameter in cosmos_guardrail/cosmos_utils.py:

#!/usr/bin/env bash
COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")"
sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py"

3. Serve Cosmos3

Serve Cosmos3-Nano directly from the Hugging Face model ID:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1

With --performance-mode auto, Cosmos3 Nano keeps its DiT and VAE resident when every selected GPU has at least 90 GiB available at startup. Other Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold, auto mode retains the conservative DiT component-offload policy. Cosmos3 runs one DiT per pipeline, so component offload above the threshold only pays to copy the weights out to host memory and back on every request. Serve Cosmos3-Super across multiple GPUs as shown below so each rank holds a shard of the weights. For Cosmos3-Super, split the model across multiple GPUs:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4

The server also accepts the specialized nvidia/Cosmos3-Super-Text2Image and nvidia/Cosmos3-Super-Image2Video checkpoint IDs.

Edge checkpoints

Cosmos3-Edge is a 4B dense model and can be served on one GPU:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1

Edge is trained for 256p and 480p generation. Its default video configuration is 832x480 with guidance_scale=5.0; its default image configuration is 640x640 with guidance_scale=7.0. Supported sizes are 832x480, 480x832, 640x480, 480x640, 480x480, 640x640, 448x256, 256x448, and 256x256. Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with nvidia/Cosmos3-Edge-Policy-DROID.

Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4

For distilled I2V, replace the model path with nvidia/Cosmos3-Super-Image2Video-4Step. SGLang detects both checkpoints from scheduler/scheduler_config.json, uses the checkpoint’s fixed four-step sigma schedule, and forces guidance_scale=1.0. Do not tune num_inference_steps or flow_shift for these checkpoints.

4. OpenAI-Compatible Requests

Text to image

Cosmos3 text-to-image uses /v1/images/generations. The default Cosmos3 image response is b64_json, matching vLLM-Omni’s examples.

Command

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

With a server running nvidia/Cosmos3-Super-Text2Image-4Step, omit the scheduler controls and use guidance_scale=1.0:

Command

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

Text to video with sound

Use /v1/videos to create an asynchronous job, then poll the job and download the completed MP4. Set generate_sound=true to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4

Image to video

This mirrors the official nvidia/Cosmos3-Nano Hugging Face image-to-video example:

Python

import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)

For the distilled I2V checkpoint, use the same API with a server running nvidia/Cosmos3-Super-Image2Video-4Step. The recommended request is 480p and does not specify scheduler controls:

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

Video to video

Upload a source video with video_reference. Cosmos3 keeps latent frames [0, 1] by default and generates the remaining frames. Use condition_frame_indexes to select different latent frames, and condition_video_keep to take conditioning frames from the start or end of the source.

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

Action generation

For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1

Use nvidia/Cosmos3-Edge-Policy-DROID in the same command to serve the smaller 4B policy checkpoint. policy and inverse_dynamics return actions, so their canonical API is the synchronous /v1/actions/generations endpoint. The following request predicts a 16-step action chunk from one observation image. action_horizon=16 maps to the model’s num_frames=17 convention.

Python

import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])

Use GET /v1/actions/metadata to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the /v1/actions/realtime websocket use the same action envelope. To batch policy observations inside one request, opt in with a bounded batch size:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1 \
  --batching-max-size 4

Send one image per observation as a list or [B, H, W, C] uint8 array in input.input_reference, and either one prompt per image or one scalar prompt to broadcast across the batch. Batched prompts must currently tokenize to the same length because Cosmos3 GEN cross-attention does not mask padded text K/V. All items in one request share the domain, resolution, action horizon, and denoise settings. The standard action envelope returns one data[i] item per input, each with action shape [H, D]. For a compact msgpack response containing one [B, H, D] array, set runtime.response_format="raw" and read the top-level actions field. For JSON, input_reference can be a list of base64 image payloads. For msgpack, it can be a packed uint8 numpy array directly:

JSON

{
  "input": {
    "prompt": ["pick up the block", "close the drawer"],
    "input_reference": [
      {"b64_json": "<first-image-base64>"},
      {"b64_json": "<second-image-base64>"}
    ]
  },
  "parameters": {
    "action_mode": "policy",
    "domain_name": "droid_lerobot"
  }
}

The batch size cannot exceed --batching-max-size; this keeps one request from bypassing the server’s configured memory limit. Batching applies to action_mode="policy" only. A request seed controls the random stream for the whole batch, so a batched result is deterministic for that request but is not expected to be bit-exact with separately seeded B=1 requests. inverse_dynamics also uses /v1/actions/generations; set action_mode="inverse_dynamics" and pass an observation video URL or server-local path as input.observation.video. Select the embodiment head with domain_name or domain_id; set raw_action_dim explicitly when it cannot be inferred from the domain name. forward_dynamics is intentionally different: it consumes an action array and predicts video, so it remains on /v1/videos. Action-producing modes submitted to /v1/videos return HTTP 400 with the canonical action endpoint in the error message.

5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as size, num_frames, fps, num_inference_steps, guidance_scale, negative_prompt, and seed. For distilled checkpoints, SGLang replaces num_inference_steps with the checkpoint’s fixed four-step schedule and forces guidance_scale=1.0; negative-prompt CFG and request-level flow_shift do not apply. Top-level Cosmos3 request fields:

max_sequence_length: maximum text token length used by the Cosmos3 tokenizer.
flow_shift: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses --flow-shift, then the mode default (3.0 for T2I, 10.0 for non-Edge video and all action modes, or 3.0 for Edge video).
guidance_interval: optional [start, end] noise interval for CFG. Non-distilled T2I defaults to [400, 1000]; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

generate_sound: generate a sound track whose duration follows num_frames / fps.
sound_duration: explicit sound duration in seconds; takes precedence over the derived duration.
condition_frame_indexes: V2V latent-frame indexes to keep from the source video; defaults to [0, 1].
condition_video_keep: use the first or last source frames for V2V conditioning.
action_mode: policy, forward_dynamics, or inverse_dynamics.
domain_name / domain_id: select the action embodiment head.
raw_action_dim: number of active action dimensions; inferred for known domain names.
action: action array with shape [T, D], required by forward_dynamics.
action_fps: action-token frame rate for temporal mRoPE; defaults to the video FPS.
action_view_point: viewpoint used in the structured action caption.
action_normalization: dataset normalization mode, such as quantile, meanstd, or minmax.

Pass model-specific controls through extra_body with the OpenAI Python SDK. Raw JSON may keep them at the top level; multipart video requests should put them in the extra_params JSON object. The legacy image extra_args container remains accepted for compatibility, but new clients should use extra_body:

use_duration_template: whether to append SGLang’s generated duration suffix to video prompts.
use_resolution_template: accepted for vLLM-Omni request compatibility.
use_system_prompt: whether to add the Cosmos3 system prompt to the chat template.
guardrails or use_guardrails: per-request guardrail toggle when the server started with guardrails enabled.

​1. Model Introduction

​2. Installation

​3. Serve Cosmos3

​Edge checkpoints

​Distilled checkpoints

​4. OpenAI-Compatible Requests

​Text to image

​Text to video with sound

​Image to video

​Video to video

​Action generation

​5. Cosmos3 Parameters

1. Model Introduction

2. Installation

3. Serve Cosmos3

Edge checkpoints

Distilled checkpoints

4. OpenAI-Compatible Requests

Text to image

Text to video with sound

Image to video

Video to video

Action generation

5. Cosmos3 Parameters