Skip to main content
When the KV cache pool is full, the radix cache reclaims space by evicting cached prefixes. The eviction policy decides which prefix goes first. The default, lru, is the right choice for most workloads; the others trade recency for hit frequency, reuse history, or request priority.

​
How eviction picks a victim

Eviction only ever considers evictable leaves: nodes whose KV is present, unlocked (no in-flight request holds them), and not shadowed by a child that still holds KV. The root is never evictable. The policy scores each candidate and the lowest score is evicted first. Once a leaf is evicted, its parent may become a leaf and re-enter the candidate set, so eviction walks a branch from its tip toward the root. A policy only scores. It does not decide how much to free, and it cannot pin KV in memory — a node protected by the policy is still evicted if reclaiming everything else is not enough.

​
Available policies

Select one with --radix-eviction-policy. All of them fall back to least-recently-used order within a tie. Notes on the scoring inputs:
  • Hit count (lfu, slru) counts how many times a node was matched by a later request. It is not incremented for chunked-prefill steps, for evicted nodes, or under write-back HiCache.
  • Request priority (priority) is the priority field of the request that inserted the prefix; a node reached by several requests keeps the highest priority among them. Without priority scheduling every node is priority 0 and this policy is equivalent to lru.
fifo, mru, and filo also exist in the policy registry but are not offered on the command line. They are reachable only by out-of-tree code that extends the choice list, and exist for experiments rather than serving.

​
Tuning a policy

Some policies take parameters. Pass them as a json object to --radix-eviction-policy-config; the keys are the policy’s own, so they are only valid for the policy you selected.
Command
Omit the flag to accept every default. An unrecognized key fails at startup rather than being ignored:
--radix-eviction-policy-config is not supported by the experimental Rust tree core (SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust), which builds its strategy from the policy name alone. Passing both fails at startup.

​
Policy parameters

Only slru currently takes a parameter. lru, lfu, and priority take none, so --radix-eviction-policy-config has no effect with them and any key is an error.

​
slru parameters

slru splits the cache into a probationary segment and a protected segment. A prefix enters probationary, and is promoted to protected once it has been hit enough times. Everything probationary is evicted before anything protected. Raising it makes promotion harder, so the protected set stays small and closer to your genuinely hot prefixes; a prefix hit three times stays probationary at 4 but is protected at the default 2. Lowering it to 1 promotes any prefix that is reused even once, which approaches lru with a one-hit grace period.

​
Which policy to pick

Start with lru and change it only against a measured cache hit rate — the counters are exposed under --enable-metrics. lfu and slru help when your traffic has a stable hot set that a recency-only policy keeps flushing; they hurt when prefix popularity shifts over time, because a prefix that earned a high hit count keeps its advantage after it stops being useful.