Skip to main content
SGLang Simulator reuses SGLang’s scheduler, request lifecycle, and KV-cache implementation while replacing model forward execution with a latency predictor. Use it to compare scheduling and cache configurations on timestamped or synthetic workloads without loading model weights.

​
Supported scope

SGLang Simulator tracks the current main branch and recent SGLang releases. The current integration is validated with v0.5.16, v0.5.17, v0.5.18, and main. The initial upstream scope uses one simulated worker with tp_size=1, ep_size=1, dp_size=1, and pp_size=1. A simulator configuration can describe a larger target system for latency prediction, but the SGLang runtime process topology remains single-worker. The simulator supports:
  • synthetic request rates, ShareGPT workloads, and timestamped Autobench traces;
  • OFFLINE logical-time simulation and BLOCKING wall-clock replay;
  • AIConfigurator, ML, and replay latency predictors;
  • SGLang prefix caching and HiCache; and
  • serving-compatible TTFT, TPOT, ITL, throughput, and cache-hit metrics.

​
Install from the SGLang repository

Use the simulator and SGLang source from the same monorepo checkout:
AIConfigurator is optional. Install the validated extra only when you use an AIConfigurator predictor:

​
Start a simulator server

Choose a fresh output directory for every run. The server owns the simulation mode and writes metrics to this directory.
OFFLINE advances the simulator’s logical clock without sleeping. BLOCKING also sleeps for predicted forward and cache-load latency, which is useful when a client must observe simulated wall-clock pacing.

​
Send a workload

In another terminal, export the same output directory and run the simulator-aware serving benchmark from the repository root:
The benchmark injects logical arrival metadata into each request and displays the server-side simulator metrics. For timestamped traffic, use the simulator-owned Autobench JSONL format and add --use-trace-timestamps.

​
Read the results

The output directory contains:
  • metrics.json: aggregate latency, throughput, and cache metrics;
  • request.jsonl: per-request timing and cache information; and
  • iteration.jsonl: scheduler batch composition and predicted iteration latency.
Use a unique output directory for each run so metrics from separate experiments are not mixed. See the SGLang Simulator source README for simulator configuration fields, predictor examples, and maintained tests.