PPLX-Decider-v1-27B

​

Deployment

pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang

docker pull lmsysorg/sglang:dev

​

Playground

​

1. Model Introduction

​

2. Configuration Tips

• Route. Send requests to /v1/systemone, the System One API the checkpoint was trained for. /v1/decisions refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
• Clients. Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See System One compatible API for the request and response reference.
• Images. Pass each image in images as base64 bytes, a base64 data URL, or an http(s) URL. They precede the text in every question, as in the checkpoint’s reference code. The checkpoint’s processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens: a 1024×768 screenshot is 234 tokens, and a 4K one is no more expensive than 1080p (252). Crop to the region that matters rather than sending a larger image.
• What a decision costs. Each question is its own prompt: an ~84-token wrapper (system message, chat template, answer instructions), the state, the question, and its options, read in one prefill pass. Several questions in one request are scored together, but each re-reads the whole state, so the cost of a request is roughly questions × state tokens.
• Prefix cache. On this hybrid model the prefix cache resumes from where an earlier prompt ended, because that is where the Gated DeltaNet state is saved, not from any shared prefix. An identical request measured 215 ms cold and 56 ms cached on GB300, but a different question about the same state, and the other questions in the same request, reused none of the state. The GB300 recipe therefore turns the cache off with --disable-radix-cache: unique-state traffic loses no reuse, each request holds one state slot instead of five, and the per-request state bookkeeping goes away, which took a short question from 59 to 51 ms and an image request from 126 to 87 ms with the attention kernel held fixed. Turn it back on only if your traffic repeats identical requests, the same state and the same question, often enough to matter.
• Prompt length. The model was fine-tuned on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves the backbone’s full 262,144-token context, so keep states within the trained length. A longer prompt is answered, but its accuracy is untested.
• Throughput is compute-bound. A decision is pure prefill, and one GB300 reads about 27,000 prompt tokens per second in BF16 once batches are full: about 71 requests per second for a 382-token question, and 3.3 for an 8K-token state. Past that point, more concurrency only adds queueing. The tables in §3 map concurrency to latency, so pick the highest concurrency that fits your latency budget.
• Memory. The weights take 52 GB, and the default --mem-fraction-static hands the rest of the GPU to the KV and Gated DeltaNet state pools. The startup log line max_running_requests is capped to ... by the mamba state cache is expected: on GB300, each request reserves five state slots with the prefix cache on (a cap of 118) and one with it off (593). Even the lower cap only binds for questions shorter than about 140 tokens, because a prefill batch fills its 16,384-token chunk first. --mamba-ssm-dtype bfloat16 halves the state pool but measured no faster, so keep the checkpoint’s FP32 state.
• GB300 kernels. With the prefix cache off, SGLang picks the TRT-LLM attention kernel with 64-token pages on GB300 (Triton with 1-token pages while the cache is on), and that kernel is where the long-state gain comes from: an 8K-token state drops from 377 to 325 ms and saturates at 3.30 instead of 2.87 requests per second, the same as forcing --attention-backend trtllm_mha --page-size 64 with the cache on. FlashInfer’s Gated DeltaNet prefill (--linear-attn-prefill-backend flashinfer) adds another 4-8% on text, taking the 8K-token state to 300 ms and 3.56 requests per second, but it made image requests slower (101 vs 87 ms alone, 9% less throughput), so the recipe leaves it to the Playground for text-only traffic. A larger prefill chunk (--chunked-prefill-size 32768) did not help: one 16,384-token chunk already keeps the GPU busy.
• More GPUs. The model fits one GPU, so add throughput with data-parallel replicas: --dp-size 2 on two GB300s served 139 short questions per second (1.96× one GPU) and 6.5 8K-token states per second, and because the questions of one request are spread over the replicas, it also cut four questions on a 1K-token state from 179 to 104 ms. The extra dispatch hop added about 15 ms to a lone short question. Tensor parallelism only pays off for latency on long states: --tp 2 took an 8K-token state from 325 to 208 ms, but delivered less throughput than two replicas and made a short question slower (68 ms), because the all-reduce dominates small batches. SGLang Model Gateway does not route /v1/systemone yet, so to spread load over several servers instead of --dp-size, use a plain HTTP load balancer.
• FP8. --quantization fp8 quantizes the BF16 weights at load time and measured about 20% more throughput on GB300 (84 vs 70 requests per second for short questions), with the same Belebele and WinoGrande scores. But its probabilities moved by 0.010 on average and up to 0.20 against the reference implementation, four times the drift of any BF16 configuration, and 0.9% of answers changed their top option. The calibrated probabilities are this model’s output, so the recipes stay in BF16, and the Playground offers FP8 for workloads that only use the top choice.
• Reproducibility. Two launches of the same recipe returned bit-identical probabilities for the same requests at the same concurrency. Against the checkpoint’s reference implementation, the largest probability difference per question averages 0.003 (see §3), because batch composition and kernels differ, and near-ties at 0.500 can flip.

​

3. Benchmarks

​

3.1 Speed

​

3.2 Accuracy and parity

​

3.3 What else was measured

import asyncio

import aiohttp
from datasets import load_dataset

URL = "http://localhost:30000/v1/systemone"


async def main():
    rows = load_dataset("facebook/belebele", "eng_Latn", split="test")
    limit = asyncio.Semaphore(32)

    async def ask(session, row):
        body = {
            "model": "perplexity-ai/pplx-decider-v1-27b",
            "state": row["flores_passage"],
            "questions": {
                "answer": {
                    "type": "choice",
                    "instructions": row["question"],
                    "criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)},
                }
            },
        }
        async with limit, session.post(URL, json=body) as response:
            response.raise_for_status()
            answer = (await response.json())["answers"]["answer"]
        return answer["choice"] == row["correct_answer_num"]

    async with aiohttp.ClientSession() as session:
        correct = await asyncio.gather(*(ask(session, row) for row in rows))
    print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%")


asyncio.run(main())

Belebele eng_Latn: 870/900 = 96.67%

import asyncio
import random
import statistics
import sys
import time

import aiohttp

BASE = "http://localhost:30000"
STATE_WORDS, CONCURRENCY, REQUESTS = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
WORDS = ("customer payment refund invoice order delivery account login error update server outage "
         "contract price plan report urgent today please help the a and with for from to of in on").split()
QUESTION = {
    "type": "choice",
    "instructions": "Which team should handle this request?",
    "criteria": {"billing": "Charges and refunds", "technical_support": "Integration errors",
                 "sales": "Questions about buying a product", "legal": "Contracts and liability"},
}


def body(rng):
    # A unique random state per request, so no request reuses another's cache.
    state = " ".join(rng.choices(WORDS, k=STATE_WORDS))
    return {"model": "perplexity-ai/pplx-decider-v1-27b", "state": state, "questions": {"q": QUESTION}}


async def main():
    rng = random.Random(0)
    bodies = [body(rng) for _ in range(REQUESTS)]
    latencies, tokens = [], 0
    async with aiohttp.ClientSession() as session:
        for item in [body(rng) for _ in range(8)]:  # warmup
            async with session.post(f"{BASE}/v1/systemone", json=item) as response:
                response.raise_for_status()
        async with session.post(f"{BASE}/flush_cache") as response:
            await response.read()
        queue = iter(bodies)

        async def worker():
            nonlocal tokens
            for item in queue:
                start = time.perf_counter()
                async with session.post(f"{BASE}/v1/systemone", json=item) as response:
                    response.raise_for_status()
                    tokens += (await response.json())["usage"]["input_tokens"]
                latencies.append(time.perf_counter() - start)

        start = time.perf_counter()
        await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))
        elapsed = time.perf_counter() - start
    print(f"{REQUESTS / elapsed:.1f} req/s, {tokens / elapsed:.0f} prompt tok/s, "
          f"P50 {1000 * statistics.median(latencies):.1f} ms, {tokens / REQUESTS:.0f} tokens/request")


asyncio.run(main())

$ python bench.py 270 1 64
19.6 req/s, 7393 prompt tok/s, P50 51.0 ms, 378 tokens/request
$ python bench.py 270 64 512
68.4 req/s, 25865 prompt tok/s, P50 899.0 ms, 378 tokens/request

​

4. Advanced Usage

​

4.1 Text Decisions

import requests

response = requests.post(
    "http://localhost:30000/v1/systemone",
    json={
        "model": "perplexity-ai/pplx-decider-v1-27b",
        "state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
        "questions": {
            "routing": {
                "type": "choice",
                "instructions": "Which team should handle this request?",
                "criteria": {
                    "billing": "Charges and refunds",
                    "technical_support": "Integration errors",
                    "sales": "Questions about buying a product",
                },
            },
            "urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
        },
    },
    timeout=60,
)
response.raise_for_status()
for name, answer in response.json()["answers"].items():
    print(name, answer)

routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.987854396908662, 'probabilities': {'billing': 0.004840538662952211, 'technical_support': 0.9919029312724412, 'sales': 0.0032565300646064375}}
urgency {'type': 'noul', 'noul': 0.9929980993021188}

​

4.2 Image Decisions

import base64
from pathlib import Path

import requests

image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
response = requests.post(
    "http://localhost:30000/v1/systemone",
    json={
        "model": "perplexity-ai/pplx-decider-v1-27b",
        "state": "Look at the supplied image.",
        "images": [image],
        "questions": {
            "color": {
                "type": "choice",
                "instructions": "What is the dominant color?",
                "criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
            }
        },
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["color"])

{'type': 'choice', 'choice': 'red', 'confidence': 0.9498445161931509, 'probabilities': {'red': 0.9623833871448632, 'green': 0.009006826727044857, 'blue': 0.00926546931819573, 'other': 0.01934431680989631}}