Nemotron3.5-Lightning - SGLang Documentation

Deployment

Install SGLang

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.

Python (pip / uv)
Docker

Command

pip install --upgrade pip
pip install uv
SGLANG_BUILD_RUST_EXTS=none uv pip install 'git+https://github.com/sgl-project/sglang.git@refs/pull/33554/head#subdirectory=python'

Then run the Python output of the command panel below in that environment.

Command

docker pull lmsysorg/sglang:dev-nemotron3-5-lightning

For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick your hardware and recipe to generate the launch command. Every platform publishes four operating points: Balanced (no speculation) plus three speculative decoders — MTP, DFlash, and DSpark. Use the Playground below to explore knobs beyond them.

Playground

The Playground is where you experiment with SGLang features beyond the verified matrix. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

1. Model Introduction

NVIDIA Nemotron 3.5 Lightning is a 30B-A3B hybrid reasoning LLM. See the Hugging Face model cards below for architecture and evaluation details.

Checkpoint	Precision	Use
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4	NVFP4	Serving — the checkpoint this page deploys
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16	BF16	Full-precision reference
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash	W4A16	DFlash speculative draft model
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark	W4A16	DSpark speculative draft model

MTP needs no separate download — the draft head is embedded in the target checkpoint.

2. Usage

The server speaks the OpenAI API. With --reasoning-parser nemotron_3 enabled, the thinking trace lands in message.reasoning_content and the answer in message.content.

Example

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="null",
)

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is SGLang?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=1024,
)
choice = response.choices[0]
print("Reasoning:", choice.message.reasoning_content)
print("Content:", choice.message.content)

2.1 Tool Calling

With --tool-call-parser qwen3_coder enabled, structured tool calls are returned in message.tool_calls.

Example

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="null",
)

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "calculate_tip",
            "parameters": {
                "type": "object",
                "properties": {
                    "bill_total": {"type": "integer", "description": "The total amount of the bill"},
                    "tip_percentage": {"type": "integer", "description": "The percentage of tip to be applied"},
                },
                "required": ["bill_total", "tip_percentage"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
    messages=[{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}],
    tools=TOOLS,
    max_tokens=1024,
)
choice = response.choices[0]
print("Content:", choice.message.content)
print("Tool calls:", choice.message.tool_calls)

​Deployment

​Playground

​1. Model Introduction

​2. Usage

​2.1 Tool Calling

Deployment

Playground

1. Model Introduction

2. Usage

2.1 Tool Calling