Skip to main content
SGLang provides opt-in Aaronson-Gumbel watermarking for sampled text. The watermark replaces random token selection with a keyed, deterministic choice that follows the same output distribution under the ideal independent-uniform hash model, so anyone holding the secret key can later test whether a token sequence carries the watermark. Detection needs only the key and the token IDs; it does not need the model or a GPU. This feature can support transparency controls such as the machine-readable marking and detectability described in Article 50 of the EU AI Act. It is one technical signal, not a complete compliance program or proof of authorship.

​
How it works

​
Generation

At each sampling step, SGLang first applies frequency, presence, and repetition penalties, logit bias, and any structured-output grammar mask, followed by the request’s temperature and its top-k, top-p, and min-p truncation, then renormalizes. Let p_i be the resulting probability of candidate token i; grammar-masked and truncated tokens have p_i = 0 and can never be selected. For each candidate, SGLang hashes the key, the preceding token context, and the token ID into a uniform value u_i in (0, 1) and selects
When the u_i are independent uniforms, this rule selects token i with probability exactly p_i, so a single step is distributed like ordinary sampling. Ties go to the lowest token ID. A position is watermarked only when all of the following hold; otherwise the request samples that position normally:
  • The request has watermarking enabled (see Per-request control).
  • The request is not greedy. temperature=0 is normalized to top_k=1, and top_k=1 positions are never watermarked.
  • The context is not empty.
  • The context has not already been watermarked earlier in the same request. Runtime identifies a context by its 32-bit context hash.
  • The request has watermarked fewer than min(4,096, per-request token-pool capacity) contexts.
  • The largest truncated probability is at most the server’s max_probability. The default 1.0 disables this gate.
Only a watermarked position enters the request’s context history. A position skipped by the probability gate therefore leaves its context eligible for a later, higher-entropy occurrence, and once a request reaches its context budget the remaining tokens use ordinary sampling. The repeated-context rule keeps a repeated context from forcing the same token indefinitely. The context for a position is the last h tokens before it, where h is the request’s context window, taken from the prompt followed by the tokens generated so far. The first generated token therefore uses the last h prompt tokens. A context is shorter than h only when fewer than h tokens precede the position.

​
Hash contract

All arithmetic wraps to unsigned 32 bits. The primitives are the MurmurHash3 x86 32-bit body mix and finalizer:
A key is 1 to 16 hexadecimal digits with an optional 0x prefix, interpreted as an unsigned 64-bit integer; key_lo and key_hi are its low and high 32-bit words. For a context of L token IDs c_1 ... c_L and a candidate token t:
The context hash covers only the context tokens; keys enter only through the token hash and, in dual-key mode, through a per-context coin that chooses which key produces the token hash:
a_lo/a_hi and b_lo/b_hi are the words of key A and key B, and P is the server’s mixing_probability.

​
Detection

A detector holding the key scores each generated token t with s = -log(1 - u), where u comes from the hash contract above using that token’s context. Without the watermark, each s is an independent Exponential(1) sample. With the watermark, the generator preferred tokens with large u, so the scores are inflated. Over n scored contexts, the score S = sum(s) follows Gamma(shape=n, scale=1) under the null hypothesis that the text is not watermarked with this key. The detector reports the upper-tail probability P[Gamma(n, 1) >= S] as the p-value, its natural logarithm (finite even when the p-value underflows to 0.0), and the normal approximation z = (S - n) / sqrt(n). To match generation, a detector must follow these rules:
  1. Use the exact prompt token IDs that preceded the completion and the request’s context window h. With prompt IDs, the first completion token’s context is the last h prompt tokens. Without prompt IDs, the first h completion tokens have unknown contexts and are skipped.
  2. Score each distinct context once. The detector compares exact context token tuples, whereas generation compares 32-bit context hashes. After a hash collision, generation skips a later distinct context that the detector still scores. That extra score is an ordinary null sample, so the mismatch can only weaken detection, never create evidence.
  3. Stop after the generation request’s context budget and report a prefix test. The default detector budget is 4,096 distinct contexts. If the server’s per-request token-pool capacity is lower, pass that lower value as max_contexts; scoring the unforced tail adds null samples that dilute the statistic without adding evidence. With a maximum-probability gate below 1.0, the model-free detector cannot identify skipped positions, so this fixed prefix is conservative and may omit signal from later forced positions.
Positions that generation left unforced (greedy positions, and positions skipped by max_probability) are indistinguishable from ordinary text to the detector, which scores them as null samples; they dilute the statistic but cannot produce false evidence. A low p-value is evidence that the text carries the keyed watermark. It is not a calibrated probability that a particular author or model produced the text. Applications must choose a decision threshold and measure false-positive and false-negative rates on their own traffic.

​
Server configuration

--enable-watermark turns the feature on; every other watermark setting lives in one JSON object passed with --watermark-config, either as a file path or inline. Restrict the file to the server account and launch a CUDA server:
Every field is optional and any other field is rejected. The config must be a JSON object of at most 4,096 bytes. A value that starts with { is parsed as inline JSON; any other value is read as a path to a regular file, and SGLang logs a warning when that file is readable by group or other users. Inline JSON is visible in the process command line, including /proc/*/cmdline on Linux, so pass keys through a file. --watermark-config requires --enable-watermark. A server key alone does not watermark every request; without default_enabled or enforce_all, requests must opt in. SGLang redacts the keys and the --watermark-config value from the recorded launch command, startup logs, request logs, server information responses, and request crash-dump payloads. Process and GPU core dumps may contain in-memory keys. Protect those artifacts, the configuration file, and the transport used by clients that send per-request keys.

​
Per-request control

The native /generate API and the OpenAI-compatible /v1/completions and /v1/chat/completions APIs accept a watermark object with three optional fields: On /generate, pass watermark either at the top level of the request or inside sampling_params, but not both; a batched request may pass one object per prompt as a list. On the OpenAI-compatible APIs, pass it as a top-level request field:
The server resolves each request as follows: The request also returns HTTP 400 when:
  • the watermark object sets neither enabled nor key, or contains an unknown field;
  • key is combined with "enabled": false;
  • a key is not 1 to 16 hexadecimal digits;
  • context_window is not a positive integer or exceeds the server maximum;
  • a watermarked request uses beam search.
Request-level keys are credentials carried in the request body. Use an authenticated TLS endpoint whenever they cross a host boundary, and avoid recording request bodies in proxies or application logs.

​
Dual-key mode

key_b in --watermark-config applies dual-key selection to every watermarked request. The request’s resolved key, whether the server key or a request key, is key A, and the server’s second key is key B. A request key therefore replaces key A while key B stays active. Each eligible context uses key A with probability mixing_probability and key B otherwise, according to the coin in the hash contract; truncation, repeated-context, and greedy rules are unchanged. Without key B, selection uses the single-key path. A detector holding both keys splits the distinct contexts by the coin, scores each context with its selected key, and tests the union, which is the strongest statistic; it can also test each partition separately. A detector holding only one key can score every distinct context with that key, but the contexts selected for the other key contribute only null samples and dilute the signal.

​
Using the detector

WatermarkDetector implements the detection rules above. Construct it with the same key, key B, mixing probability, and context window that generation used: A minimal reference detection server exposes the in-tree detector over HTTP and reads keys only from --watermark-config.
  • detect_tokens(token_ids, *, prompt_token_ids=()) scores completion token IDs.
  • detect_text(text, tokenizer, *, prompt_token_ids=()) first encodes the text with tokenizer.encode(text, add_special_tokens=False). Any object with that method works, including Hugging Face tokenizers.
  • detect(token_ids, *, key, key_b=None, prompt_token_ids=(), mixing_probability=0.5, context_window=4) is a one-call shortcut.
Token IDs must be unsigned 32-bit integers; any iterable of Python or NumPy integers is accepted. Every call returns a WatermarkDetection: Each statistic is a WatermarkStatistics with num_contexts, score, z_score, p_value, and log_p_value. Prefer the server’s token IDs to re-tokenized text, because re-encoding decoded text does not always reproduce the generated IDs. With "return_token_ids": true, non-streaming chat completions return prompt_token_ids and response_token_ids on each choice, and completions return prompt_token_ids and token_ids.

​
In an SGLang environment

To score text, pass the model’s tokenizer and, when available, the prompt IDs:
For a dual-key server, pass both keys and the server’s mixing probability:

​
Standalone, without SGLang or PyTorch

detector.py and config.py import only the Python standard library and msgspec, and the package’s __init__.py imports only the detector. NumPy arrays are accepted as token-ID iterables, but NumPy is not required. To run detection in a separate application, copy the package directory and install msgspec:
The copy also contains core.py, the server runtime, which imports PyTorch; nothing on the detector import path loads it. Run the application with detector_app on the import path and import the detector from the copied package:
Text detection needs only an object with encode(text, *, add_special_tokens) that returns a list of IDs. For example, with the lightweight tokenizers package:
A detector written from scratch instead of reusing this package must implement the hash contract and all three detection rules: prompt alignment, exact-tuple deduplication, and the 4,096-context prefix.

​
Compatibility

In speculative decoding, verification clones the target logits before forcing, so target-model log-probability computation still sees the pre-watermark distribution. Each draft node gets its own context, rebuilt from its ancestors in the verify tree. Repeated-context admission is conservative across the whole tree: once a context hash appears in an earlier row, later rows with the same hash are not forced, even on sibling branches. This keeps duplicate entries out of the history, but reconvergent wide trees may carry less signal.

​
Limitations and performance

  • Greedy requests carry no signal, and very low-entropy distributions leave few meaningful alternatives for the selector.
  • A max_probability value below 1 keeps ordinary selection at low-entropy positions and can improve speculative draft acceptance, but the unforced positions dilute the detector statistic.
  • Short completions provide few distinct contexts and therefore weak statistical power.
  • Paraphrasing, rewriting, translation, and re-tokenization can weaken or remove the signal.
  • Detection needs the matching key or keys, mixing probability, context window, and tokenizer, and is strongest with the exact prompt token IDs.
  • A request watermarks at most min(4,096, per-request token-pool capacity) distinct contexts; later tokens use ordinary sampling.
  • The hashes are MurmurHash3, a keyed but non-cryptographic mix. There is no indistinguishability or unforgeability guarantee, and anyone holding the key can both detect and forge the watermark.
  • Speculative decoding can accept fewer draft tokens when a draft proposal does not match the watermarked target token. The impact depends on tree width and workload.
  • Mixed batches choose the selector path from the batch-wide maximum top-k. A transient unlimited-top-k request can keep that batch on the full-sort path until its sampling metadata is rebuilt.
  • Retraction reconstructs a conservative distinct-context history. Repeated retractions can reduce later signal and add scheduler-side replay work.
On one single-GPU node of each type with public Qwen/Qwen3-8B, TP1, TRT-LLM MHA, ordinary decoding, temperature 1.0, top-p 0.95, unlimited top-k, and 800-token sampled outputs, the fused selector measured the following mean output throughput over three repetitions per cell: The GB300 concurrency-eight row was remeasured after the score-precision fix on a default-off server by comparing omitted requests with request-level opt-in. These measurements describe one model, GPU, sampling configuration, and output length. Benchmark your deployment before setting capacity targets. Finite-top-k selector latency at batch size one is also hardware-dependent: in one microbenchmark shape (top_k=20), the fast path was 58.8% slower than a full sort on H200 with bfloat16 logits and 6.4% faster on GB300 with float32 logits, and at batch size eight it was faster on both. On a 2x GB300 node with public Qwen/Qwen3.5-9B, TP2, NEXTN 3/1/4, concurrency 8, 1,024-token random inputs, 512-token outputs, temperature 0.8, top-p 0.95, and top-k 20, the fused speculative path measured the following means over three repetitions: Each speculative row still requires keyed selection, but tree-context reconstruction, repeated-context admission, accepted-context recording, and ring-buffer append run as fused GPU operations. The reported cost remains workload- and acceptance-dependent.

​
See also

The fused Triton selector is adapted from Meta’s Apache-2.0 TextSeal implementation and re-implemented on the hash contract above, so detection follows this page rather than TextSeal’s own tooling. The runtime and reference detector live in python/sglang/srt/sampling/watermarking/. Registered coverage lives in test/registered/kernels/ops/sampling/test_watermark_*.py and test/registered/e2e/openai_server/test_watermark.py.