Dots3-Note - SGLang Documentation

Deployment

Install SGLang

dots3.note support is in SGLang PR #33829. Until that PR is included in a tagged SGLang release, install from a build that contains the PR.

Python (pip / uv)
Docker

Command

pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

git clone https://github.com/sgl-project/sglang.git
cd sglang
git fetch origin pull/33829/head && git checkout FETCH_HEAD
uv pip install -e python

Then run the Python output of the command panel below in that environment.

Command

docker pull lmsysorg/sglang:dev-dots3-note

This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself.For how to launch the image, see Install → Method 3: Using Docker. Substitute the inner sglang serve ... with what the command generator below produces.

Pick the checkpoint precision — the only deployment choice. The recipe runs on a single 8-GPU H200 node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet. Precision — selects the MoE path, not just the weights. The BF16 cells pin --moe-runner-backend deep_gemm with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via SGLANG_ENABLE_JIT_DEEPGEMM=1). The FP8 cells leave both at auto and let SGLang resolve the runner from the checkpoint’s quantization config. Spec Decode — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3’s MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path.

Every cell in the Deploy panel above is currently unverified: the recipe runs, but no serving round on public weights has landed (the checkpoint is not yet released). Treat the cells as starting points and re-measure throughput and accuracy on your workload.

1. Model Introduction

dots3.note is RedNote’s native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input.

Native multimodality — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor.
Native video pipeline — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path.
Hybrid attention — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers.
MTP speculative decoding — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head.

The dots3.note checkpoint is not yet publicly released. The recipes on this page were validated against SGLang PR #33829; a Hugging Face repository will be linked here at launch.

Resources: SGLang PR #33829

2. Configuration Tips

Hybrid KV pool. dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with --swa-full-tokens-ratio 0.03 setting the ratio of SWA-layer KV tokens to full-layer KV tokens (swa_tokens ≈ full_tokens × ratio). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck. MoE runner. Leave the runner at the cell default: deep_gemm for BF16 checkpoints, auto for quantized ones. DeepEP is the all-to-all transport in every cell (--moe-a2a-backend deepep, dispatch tokens per rank tuned via SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128). Attention backend. FA3 across the board: prefill, decode, and draft (--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3) with --page-size 64. MTP target verification uses FA3’s absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode. DSA. DSA indexing on full-attention layers is on by default. To disable it, add --json-model-override-args '{"index_topk":null}'. CUDA graphs. The cells enable decode-side CUDA graphs only (--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to --cuda-graph-backend-decode disabled (and expect --deepep-mode normal to be the better fit). Context length. --context-length 524288 is the model’s window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool. Language-only mode. Add --language-only to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see EPD below.

3. Advanced Usage

3.1 Native video input

dots3.note accepts a native video_url. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question.

Video Example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="dots3.note",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {"url": "https://example.com/sample.mp4"},
                },
                {"type": "text", "text": "Summarize what happens in this video."},
            ],
        }
    ],
    extra_body={
        "seq": 131072,
        "audio_cap": 0.5,
        "audio_sr": 16000,
        "k_mode": "eval_ek",
    },
)

print(response.choices[0].message.content)

Example Output

Output

Pending update...

Per-request video preprocessing controls (all optional, passed via extra_body):

Field	Default	Purpose
`seq`	`131072`	Total sequence budget used by the video flattener.
`audio_cap`	`1.0`	Maximum fraction of the input budget assigned to audio; `0` disables audio processing.
`audio_sr`	`16000`	Audio sample rate.
`k_mode`	`eval_ek`	Deterministic evaluation/sampling mode of the flattener.

These controls are request-scoped so that evaluation jobs with different context budgets can share one server. The flattener reserves room for max_new_tokens inside the budget and falls back to visual-only processing if audio would exceed the configured token budget.

Native video currently supports one video per request, and a native video cannot be mixed with separate image or audio inputs in the same request.

3.2 Image and audio input

Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format and SGLang’s multimodal serving (--enable-multimodal is in every cell). The vision and audio towers run in-process, so no extra server is needed.

3.3 Tool Calling

The cells launch with --tool-call-parser dots, so structured tool calls surface via message.tool_calls out of the box.

Tool Calling Example (Python)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]
resp = client.chat.completions.create(
    model="dots3.note",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools,
)
print(resp.choices[0].message.tool_calls)

Example Output

Output

Pending update...

3.4 Encoder/LLM Disaggregation (EPD)

Dot3NoteForCausalLM supports both roles of an encoder/LLM-disaggregated deployment:

Encoder role — serve with --encoder-only; the instance runs only the vision and audio towers.
Language role — serve with --language-only; the instance skips tower construction, leaving the memory to the language model.

See the EPD guide for how to wire the roles together.

​Deployment

​1. Model Introduction

​2. Configuration Tips

​3. Advanced Usage

​3.1 Native video input

​3.2 Image and audio input

​3.3 Tool Calling

​3.4 Encoder/LLM Disaggregation (EPD)

Deployment

1. Model Introduction

2. Configuration Tips

3. Advanced Usage

3.1 Native video input

3.2 Image and audio input

3.3 Tool Calling

3.4 Encoder/LLM Disaggregation (EPD)