Granite 4.2 - SGLang Documentation

Deployment

Install SGLang

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.

Python (pip / uv)
Docker

Command

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang

Then run the Python output of the command panel below in that environment.

Command

docker pull lmsysorg/sglang:dev

For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick a Granite 4.2 checkpoint to generate the launch command. The verified matrix covers BF16 serving on one NVIDIA H200 or B200 with tensor parallelism 1.

The H200 speed results use lmsysorg/sglang:dev at SGLang commit d59c1ddf7 and the B200 results at commit d10a656ad8; the launch recipes were verified end to end on both GPUs against the release checkpoints. Each speed point uses 80 fixed-length random requests at 8,192 input and 1,024 output tokens, 8 warmup requests, a flushed cache, greedy sampling, and ignore-EOS.

Playground

The Playground layers SGLang features on top of the verified recipe. Any override changes the badge to Not Verified until that exact configuration is tested end to end.

1. Model introduction

Granite 4.2 is IBM’s dense decoder-only language model family with 3B, 8B, and 30B checkpoints. Each checkpoint uses BF16 weights, has a configured context length of 131,072 tokens, and supports default thinking, non-thinking, low-effort thinking, and structured tool calls through its chat template. The repositories declare the Apache-2.0 license.

Variant	Total params	Position in family
Granite 4.2 3B	3B	Smallest checkpoint
Granite 4.2 8B	8B	Mid-size checkpoint
Granite 4.2 30B	30B	Largest checkpoint

Recommended generation: IBM recommends temperature=1.0 and top_p=0.95 for general chat, reasoning, and tool calling. The release checkpoints ship these values in generation_config.json; send them per request when you want to be explicit. Resources: Granite 4.2 3B · Granite 4.2 8B · Granite 4.2 30B.

2. Configuration tips

Thinking is enabled by default. Set chat_template_kwargs.enable_thinking to false for a direct answer. Set enable_thinking and low_effort to true for a shorter reasoning trace.
Give thinking enough tokens. At temperature=1.0 the default thinking mode can run past 1,000 tokens on multi-step problems. Use a max_tokens of at least 2,048 for thinking requests so the answer is not cut off.
Reasoning parser. Launch with --reasoning-parser auto, which resolves to nemotron_3 for these checkpoints, so OpenAI-compatible responses separate the trace into message.reasoning_content and the answer into message.content. Without a parser flag the reasoning markup stays inline in message.content.
Tool-call parser. Launch with --tool-call-parser auto, which resolves to qwen3_coder for these checkpoints, so tool requests are returned through message.tool_calls. Without it, raw <tool_call> markup stays in message.content.
Single-GPU sizing. All three BF16 checkpoints loaded and completed chat requests with --tp 1 --mem-fraction-static 0.8 on one H200 and on one B200. Increase TP only after validating the new topology.
Image selection. The validated path uses lmsysorg/sglang:dev. A stable image tested during validation had an incompatible dependency set before model loading, so use the recipe’s image until a newer tagged release is confirmed.

3. Advanced usage

The outputs below are verbatim captures from Granite 4.2 3B on the verified server. Sampling is stochastic, so a repeated request can produce different wording.

3.1 Thinking modes

The nemotron_3 reasoning parser keeps reasoning and final content in separate fields. Granite 4.2 accepts three chat-template modes: default thinking, non-thinking, and low-effort thinking.

Thinking modes example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
model = "ibm-granite/granite-4.2-3b"

modes = {
    "thinking": {"enable_thinking": True},
    "non-thinking": {"enable_thinking": False},
    "low-effort": {"enable_thinking": True, "low_effort": True},
}

for name, chat_template_kwargs in modes.items():
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "user", "content": "What is 17 * 23? Answer briefly."}
        ],
        extra_body={"chat_template_kwargs": chat_template_kwargs},
    )
    message = response.choices[0].message
    print(f"[{name}]")
    print("Reasoning:", getattr(message, "reasoning_content", None))
    print("Answer:", message.content)

Example output

Output

[thinking]
Reasoning: Okay, the user asked "What is 17 * 23? Answer briefly." I need to calculate 17 multiplied by 23.

Let me do the multiplication. 17 times 23.

I can break it down: 17 * 20 = 340, and 17 * 3 = 51. Then add them: 340 + 51 = 391.

Alternatively, 23 * 17: 23*10=230, 23*7=161, 230+161=391. Same result.

So the answer is 391.

The user wants a brief answer, so just state the number.

Answer:
391
[non-thinking]
Reasoning: None
Answer: 391
[low-effort]
Reasoning: Compute 17*23 = 17*20=340, plus 17*3=51 => 391.

Answer:
391

3.2 Tool calling

The qwen3_coder parser converts the model’s tool markup into OpenAI-compatible structured calls.

Tool calling example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "The city name"},
            },
            "required": ["city"],
        },
    },
}]

response = client.chat.completions.create(
    model="ibm-granite/granite-4.2-3b",
    messages=[{"role": "user", "content": "What is the weather in Boston right now?"}],
    tools=tools,
    tool_choice="auto",
)

choice = response.choices[0]
message = choice.message
print("Reasoning:", getattr(message, "reasoning_content", None))
print("Content:", message.content)
for call in message.tool_calls or []:
    print("Tool:", call.function.name)
    print("Arguments:", call.function.arguments)
print("Finish reason:", choice.finish_reason)

Example output

Output

Reasoning: Okay, the user is asking for the weather in Boston right now. I need to use the available tool called get_weather. The tool requires the city parameter. Since the user specified Boston, I'll call get_weather with city set to Boston.

Content: None
Tool: get_weather
Arguments: {"city": "Boston"}
Finish reason: tool_calls

​Deployment

​Playground

​1. Model introduction

​2. Configuration tips

​3. Advanced usage

​3.1 Thinking modes

​3.2 Tool calling

Deployment

Playground

1. Model introduction

2. Configuration tips

3. Advanced usage

3.1 Thinking modes

3.2 Tool calling