Skip to main content

Overview

CUDA graph capture (kernel warmup, attention-metadata setup and one capture per batch size) is a large part of a server’s cold start. With --cuda-graph-persistence, Foundry saves the captured graphs, together with the memory layout they reference, on a first start (save) and rebuilds them from that archive on later starts (load) instead of capturing. Foundry runs a CUDA driver hook in the scheduler processes that places every device allocation at the same address on save and load.

Installation

Foundry is an optional dependency, published on PyPI as foundry-core (import name foundry):
The PyPI wheel (Linux x86_64) is built against the torch SGLang pins. For another torch / CUDA pair, add the pair’s package index (whl/<cuda>/torch<A.B>/, see the Foundry README):

Usage

Engines with --cuda-graph-persistence run with TORCHDYNAMO_DISABLE=1. Foundry pins it for the scheduler processes (a contrary value disables persistence, see below): with TorchDynamo on, save fails at the first @torch.compile helper inside the capture window on torch 2.14 and later. On the validated models outputs are identical; a model whose MoE activation is a torch.compile fusion (gpt-oss) decodes up to ~9% slower under the pin. --cuda-graph-persistence-config names an optional Foundry TOML (workspace_root for the archive directory, base_addr / region_size for the address region, scratch_space_size); its mode key is ignored, the flag’s mode wins:
region_size is the address range every device allocation of a scheduler lands in after Foundry’s setup: weights, KV pool, graph memory and symmetric-memory reservations under 2 GiB. Only address space is reserved up front; memory is mapped per allocation. You size it: it must cover what one GPU allocates, which at the default --mem-fraction-static is about the GPU’s memory. The default is 512GB. save and load must use the same model, flags, parallel layout and GPU type; remove the archive before a save of another configuration. The launcher and every scheduler log one activation line, and load logs the restore per runner:
Run engines with and without --cuda-graph-persistence in separate processes.

What it changes

  • Decode graphs use the full backend; prefill graphs are full or disabled (default disabled). Other backends are not supported. With decode graphs disabled there is nothing to persist: the flag has no effect and Foundry is not loaded.
  • FlashInfer autotuning is off on both sides (--disable-flashinfer-autotune): tuning allocates timing-dependently inside the recorded memory layout, and save and load must select the same kernels. This covers every autotuned path in SGLang, so a kernel that would tune itself at run time uses its default tactic instead. To get tuned tactics for the model’s FlashInfer MoE / GEMM kernels, start the model once without --cuda-graph-persistence (same model and flags): it writes the autotune cache file under $SGLANG_CACHE_DIR/flashinfer/autotune/. save archives those tactics and load installs them and tunes nothing; without that file the kernels use FlashInfer’s default tactics and save logs a warning.
  • Flags and environment variables that select state a restored graph cannot replay (custom all-reduce IPC buffers, NCCL symmetric memory / NVLS / buffer registration, MSCCL++, two-batch overlap, the memory saver, graph-pool borrow / precarve, the metadata glue graph, the DCP push-combine all-reduce) are pinned; each pin prints a [Foundry] pin: line, and an explicit contrary value is not supported.
  • The graph backends and the autotune flag above also print a [Foundry] pin: line each when --cuda-graph-persistence changes them.
  • Speculative decoding is supported for EAGLE / EAGLE3 (target-verify, draft-decode and draft-extend graphs; not adaptive or multi-layer EAGLE) and for the verify-only algorithms DFLASH, NGRAM and DSPARK (target-verify and draft-verify graphs; DSPARK with SGLANG_DSPARK_USE_GREEDY_DRAFT=1, since its sampling draft allocates inside the first capture); other speculative algorithms, LoRA, elastic-EP graph recapture, unified memory (--enable-unified-memory), post-capture KV sizing and graph variants (attention variants, sharded-verify logits variants) are not supported.

Unsupported configurations

A configuration Foundry does not support still runs: Foundry logs the step and the reason, persistence is disabled and the server initialises natively (normal graph capture).
Foundry detects this while resolving the server args, at the first capture (a MoE dispatcher without prebuild(), or with its state still pending) and on save at each capture (state created inside the graph capture). A save that falls back marks its archive invalid, and a load of it also runs natively. Once load starts restoring graphs, a Foundry failure is an error. A memory failure at startup is not a fallback either: the scheduler exits and the log names the cause.
The first two are followed by the CUDA out-of-memory error of the allocation that failed.