{}

Hardware Platform

{hwGroups.map(g =>

{g.label &&

{g.label}

}

{g.items.map(item => renderButton(item, "hw", sel.hw))} {Array.from({ length: maxHwCols - g.items.length }).map((_, i) =>

)}

{matchDimSpecs.filter(d => rowVisible(d, sel)).map(d =>

{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}

)} {overlayDimSpecs.filter(d => rowVisible(d, sel)).map(d =>

{renderFlatSection(d.title, visibleOptions(d, sel), d.id, sel[d.id])}

)} {}

Command:

{cell && cell.redirect ? cell.warn &&

⚠️ {renderWarn(cell.warn)}

: <>

{VERIFY_LABEL[verifyStatus]}

{runModes.map((mode, index) => setRunMode(mode)} role="tab" aria-selected={activeRunMode === mode}> {mode === "docker" ? "Docker" : "Python"} )}

{command}

{cell && cell.warn &&

⚠️ {renderWarn(cell.warn)}

} {mtpHint &&

⚠️ Speculative decoding ({specAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency.

} {specPinnedHint &&

ℹ️ Speculative decoding ({specAlgoName}) is on and this recipe pins --max-running-requests to {specMrrValue}. Adjust it to match your target concurrency — if you remove the flag, SGLang falls back to 48.

} }

{} {benchmarks && cell && renderBenchmarkCard(benchEntry)} {} {config.showPlaygroundLink !== false &&

Need to go beyond the verified matrix?

} {} {modal === "curl" &&

setModal(null)}>

e.stopPropagation()}>

cURL example

Model: {modelName || "(unresolved)"}

{curlText}

Edit CURL_HOST / CURL_PORT in the Env panel.

} {} {modal === "env" &&

setModal(null)}>

e.stopPropagation()}>

Env / placeholder values

{placeholderGroups.curl.length > 0 &&

cURL placeholders

{placeholderGroups.curl.map(({key, label}) =>

{label} {`{{${key}}}`} setEnvDraft({ ...envDraft, [key]: e.target.value })} />

)}

} {placeholderGroups.command.length > 0 &&

Command placeholders

{placeholderGroups.command.map(({key, label}) =>

{label} {`{{${key}}}`} setEnvDraft({ ...envDraft, [key]: e.target.value })} />

)}

}

Values persist in localStorage and are reused the next time you visit any cookbook.

} {} {modal === "bench" && benchEntry && (() => { const bc = buildBenchCommands(benchEntry, sel); if (!bc) return null; const selSummary = `${sel.hw.toUpperCase()} · ${sel.variant} · ${sel.quant.toUpperCase()} · ${sel.strategy} · ${sel.nodes}`; let selConc = null; let speedCmd = null; if (bc.speed) { selConc = bc.speed.concurrencies.includes(benchConc) ? benchConc : bc.speed.concurrencies[0]; const w = bc.speed.workload; speedCmd = interpolate(bc.speed.template, { ...env, DATASET: w.dataset, ISL: w.isl, OSL: w.osl, MAX_CONCURRENCY: selConc, NUM_PROMPTS: bc.speed.numPromptsOf(selConc) }, modelName); } let selAcc = null; let accCmd = null; if (bc.accuracy.length > 0) { selAcc = bc.accuracy.find(a => a.key === benchAcc) || bc.accuracy[0]; accCmd = interpolate(selAcc.template, env, modelName); } return

setModal(null)}>

e.stopPropagation()}>

Benchmark commands

For {selSummary}. Start the server with the Deploy command above, then run these against it.

{selAcc &&

Accuracy

{bc.accuracy.length > 1 &&

benchmark: {bc.accuracy.map(a => )}

}

{selAcc.label}

{accCmd}

{bc.accuracy.length > 1 &&

Switch the benchmark chip to see each eval's command.

}

} {bc.speed &&

Speed

{bc.speed.concurrencies.length > 1 &&

max-concurrency: {bc.speed.concurrencies.map(c => )}

}

max-concurrency = {selConc}

{speedCmd}

One command — switch the concurrency chip (or edit --max-concurrency) to reproduce each Speed column.

}

Edit CURL_HOST / CURL_PORT in the Env panel.

; })()}

; }; ## Deployment dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR. ```bash Command theme={null} pip install -U uv uv venv --python 3.12 && source .venv/bin/activate git clone https://github.com/sgl-project/sglang.git cd sglang git fetch origin pull/33829/head && git checkout FETCH_HEAD uv pip install -e python ``` Then run the **Python** output of the command panel below in that environment. ```bash Command theme={null} docker pull lmsysorg/sglang:dev-dots3-note ``` This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. Pick the checkpoint precision — the only deployment choice. The recipe runs on a single 8-GPU H200 node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet. **Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config. **Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path. Every cell in the Deploy panel above is currently **unverified**: the recipe runs, but no serving round on public weights has landed (the checkpoint is not yet released). Treat the cells as starting points and re-measure throughput and accuracy on your workload. ## 1. Model Introduction dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input. * **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor. * **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path. * **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers. * **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head. The dots3.note checkpoint is **not yet publicly released**. The recipes on this page were validated against [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829); a Hugging Face repository will be linked here at launch. **Resources:** [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829) ## 2. Configuration Tips **Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck. **MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`). **Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode. **DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`. **CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit). **Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool. **Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below. ## 3. Advanced Usage ### 3.1 Native video input dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") response = client.chat.completions.create( model="dots3.note", messages=[ { "role": "user", "content": [ { "type": "video_url", "video_url": {"url": "https://example.com/sample.mp4"}, }, {"type": "text", "text": "Summarize what happens in this video."}, ], } ], extra_body={ "seq": 131072, "audio_cap": 0.5, "audio_sr": 16000, "k_mode": "eval_ek", }, ) print(response.choices[0].message.content) ``` ```text Output theme={null} Pending update... ``` Per-request video preprocessing controls (all optional, passed via `extra_body`): | Field | Default | Purpose | | ----------- | --------- | -------------------------------------------------------------------------------------- | | `seq` | `131072` | Total sequence budget used by the video flattener. | | `audio_cap` | `1.0` | Maximum fraction of the input budget assigned to audio; `0` disables audio processing. | | `audio_sr` | `16000` | Audio sample rate. | | `k_mode` | `eval_ek` | Deterministic evaluation/sampling mode of the flattener. | These controls are request-scoped so that evaluation jobs with different context budgets can share one server. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget. Native video currently supports one video per request, and a native video cannot be mixed with separate image or audio inputs in the same request. ### 3.2 Image and audio input Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format and SGLang's multimodal serving (`--enable-multimodal` is in every cell). The vision and audio towers run in-process, so no extra server is needed. ### 3.3 Tool Calling The cells launch with `--tool-call-parser dots`, so structured tool calls surface via `message.tool_calls` out of the box. ```python Example theme={null} from openai import OpenAI client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] resp = client.chat.completions.create( model="dots3.note", messages=[{"role": "user", "content": "What's the weather in Beijing?"}], tools=tools, ) print(resp.choices[0].message.tool_calls) ``` ```text Output theme={null} Pending update... ``` ### 3.4 Encoder/LLM Disaggregation (EPD) `Dot3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment: * **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers. * **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together.