--engram-config.
Every choice below returns the same rows; only memory and speed change.
Usage
--engram-config takes a JSON object. Each key answers one question, and omitted keys keep their
defaults:
The configuration is logged at startup as
DeepSeek-V4.1 Engram tables: ..., and the lookup
auto resolves to, once the model is built, as Engram lookup auto -> ....
Choosing a configuration
- Small TP groups that are short of KV cache (TP2, TP4, DP attention over 2 GPUs): use
{"placement": "cpu"}. On GPU, each rank would hold half or a quarter of the 189 GiB. - Wide TP groups and memory to spare (8 GPUs and more, including DP attention across nodes): the default. The shard per GPU is small, and over NVLink each rank reads its rows in place, with no collective.
How auto picks
With the tables in GPU memory, CUDA, and a TP group that is one NVLink domain (one node with
full NVLink, or one multi-node NVLink fabric such as GB300 NVL72), auto reads the rows in
place, as peer_read does, and logs Engram lookup auto -> peer_read. Measured with
one Engram layer, it was faster than every collective, for prefill (1K to 32K tokens per rank)
and decode (1 to 1,024 tokens per rank, in CUDA graphs):
- DP attention 8 on GB300 (two nodes), B300 and H200: 2.8-7.5x faster than
all_to_all. - DP attention 16 and 32 on GB300: 6-75x faster than the all-reduce and reduce-scatter, the only collectives there.
- Plain TP 2 and 8 on B300, TP 8 on GB300: up to 3.7x faster than the all-reduce, and never slower.
all_reduce: every rank looks up the rows it holds for all tokens, zero for the rest, and an all-reduce sums the buffers; each rank keeps its own slice.reduce_scatter: the same lookup, summed with a reduce-scatter that hands each rank only its own tokens’ rows.all_to_all: each rank looks up only the hash columns it holds, for all tokens, and sends each rank its part. It moves1 / tp_sizeof the bytes the other two move, and needs the 24 hash columns to divide over the TP group (DP 2, 3, 4, 6, 8 and 12).
auto uses all_to_all
whenever the columns divide, for prefill and decode batches alike, in CUDA graphs too; only a
prefill batch of uneven per-rank counts captured in a CUDA graph keeps the earlier collective.
Measured on GB300, B300 and H200 at DP 2-8 with host tables:
- Prefill, 512 to 65,536 tokens per rank, uniform or skewed:
all_to_allwas the fastest of the three in 134 of 144 shapes, and in all 96 from 8,192 tokens per rank up, where it was 8-62% faster than the all-reduce. Below that, another choice was ahead by at most 85 us. - Decode under CUDA graphs, 16 to 1,280 tokens per rank: from DP 4,
all_to_allwas up to 48% faster than the reduce-scatter decode used before, and at most 12 us behind at 64 tokens per rank or fewer; at DP 2 it was at most 20 us behind and up to 53 us ahead.
all_reduce keeps the collectives used before auto (an
all-reduce for prefill, a reduce-scatter for decode), and an explicit reduce_scatter or
all_to_all applies to prefill and decode alike. Without DP attention the lookup is always an
all-reduce.
Requirements and limits
placementcpu: host RAM for the whole table across the node’s ranks, and a longer load. Withreplicationnone, each rank maps its shard with 2 MiB transparent huge pages and logs the coverage it got; without them, lookups are several times slower.replicationper_node: one shared memory file per TP group, which the other ranks open through/proc. The TP group must sit on one node (--nnodes 1) and the ranks must share a PID namespace. This mapping usually gets no huge pages unless shmem transparent huge pages are on.lookupreduce_scatterorall_to_all: DP attention on CUDA;all_to_allalso needs the hash columns to divide over--tp-size.lookuppeer_read: CUDA with peer access between the GPUs (POSIX handles within a node, fabric handles across an NVLink domain);autotakes it only over NVLink. Each GPU holds1 / tp_sizeof the table. With TP1 there is no peer, and the setting has no effect.--engram-configapplies to DeepSeek-V4.1 only; a DeepSeek-V4 checkpoint without Engram layers rejects it.
Deprecated environment variables
Earlier releases chose the placement with environment variables. They still work when--engram-config is not given, with a deprecation warning that prints the equivalent JSON, and are
ignored when it is:
SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT used to default to shared; a host table set by
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 alone now gets one shard per rank, which is backed by
huge pages.