Skip to main content
DeepSeek-V4.1 adds an Engram memory to two of its layers: the preceding tokens are hashed into two large static FP8 tables, and the rows they select are gated into the hidden state. The tables are only ever looked up, never computed over, and for DeepSeek-V4.1-Flash they hold about 189 GiB together, more than any other block of weights. Where they live decides how much memory is left for the KV cache, so SGLang lets you choose it with one argument, --engram-config. Every choice below returns the same rows; only memory and speed change.

​
Usage

--engram-config takes a JSON object. Each key answers one question, and omitted keys keep their defaults: The resolved configuration is logged at startup as DeepSeek-V4.1 Engram tables: ....

​
Choosing a configuration

  • Small TP groups that are short of KV cache (TP2, TP4, DP attention over 2 GPUs): use {"placement": "cpu"}. On GPU, each rank would hold half or a quarter of the 189 GiB.
  • Wide TP groups (8 GPUs and more, including DP attention across nodes): the shard per GPU is small, and {"lookup": "peer_read"} removes the collective that each Engram layer otherwise runs per step, a cost that grows with the group size.
  • Latency-bound serving with memory to spare: the default keeps every lookup in device memory.

​
Requirements and limits

  • placement cpu: host RAM for the whole table across the node’s ranks, and a longer load. With replication none, each rank maps its shard with 2 MiB transparent huge pages and logs the coverage it got; without them, lookups are several times slower.
  • replication per_node: one shared memory file per TP group, which the other ranks open through /proc. The TP group must sit on one node (--nnodes 1) and the ranks must share a PID namespace. This mapping usually gets no huge pages unless shmem transparent huge pages are on.
  • lookup peer_read: CUDA with peer access between the GPUs (POSIX handles within a node, fabric handles across an NVLink domain). Each GPU holds 1 / tp_size of the table. With TP1 there is no peer, and the setting has no effect.
  • --engram-config applies to DeepSeek-V4.1 only; a DeepSeek-V4 checkpoint without Engram layers rejects it.

​
Deprecated environment variables

Earlier releases chose the placement with environment variables. They still work when --engram-config is not given, with a deprecation warning that prints the equivalent JSON, and are ignored when it is: SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT used to default to shared; a host table set by SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 alone now gets one shard per rank, which is backed by huge pages.