Skip to main content
HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small “hot” KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.
Prerequisites: HiSparse works with models that use DeepSeek Sparse Attention (DSA) architectures (e.g., DeepSeek-V3.2, GLM-5.1) and DeepSeek V4. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires PD disaggregation mode and is enabled on the decode instance only.

​
Why HiSparse?

In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by:
  • Reducing GPU memory per request: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length.
  • On-demand swap-in: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores.
  • Transparent to prefill: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes.

​
Design Overview

​
Decode Workflow

Each decode step follows this flow:
  1. Forward decode — generate the next token
  2. Top-k selection — select the most relevant token positions via attention scores
  3. Swap-in — the CUDA kernel loads top-k KV entries from host to device buffer:
    • Short sequences (seq_len ≤ device_buffer_size): fast path, all KV already in buffer
    • Long sequences: hit detection → LRU reordering → miss handling (host → device copy)
  4. Decode attention — compute attention using the top-k device locations
  5. Eager backup — asynchronously copy the previous token’s KV from device to host

​
PD Disaggregation Integration (Direct-to-Host)

In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance’s host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step.
For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4_indexer and C128 KV remain device-to-device transfers.

​
Server Arguments

ArgumentType / DefaultDescription
—enable-hisparseflag; default: disabledEnable HiSparse on the decode instance
—hisparse-configJSON stringConfiguration for HiSparse (see below)

​
HiSparse Config Parameters

Pass as a JSON string via --hisparse-config:
ParameterType / DefaultDescription
top_kintNumber of topk entries
device_buffer_sizeintNumber of token slots in the per-request GPU device buffer
host_to_device_ratiointRatio of logical pool size to device pool size, determining host memory capacity
swap_in_block_sizeint / 960CUDA thread-block size for the HiSparse swap-in kernel
Example: --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'

​
Shared-index prefetch (automatic)

When a model reuses one anchor layer’s top-k selection across a run of subsequent “skip” layers (DSA index_topk_freq / index_topk_pattern; native in GLM-5.2 as IndexShare), the working set of every skip layer is known the moment the anchor’s index is computed. HiSparse exploits this automatically: the anchor’s swap-in kernel records its miss plan (which host slots go to which device-buffer slots), and each skip layer replays that plan with a copy-only kernel issued ahead on a side stream, so the skip layers’ host→device IO overlaps the intervening layers’ compute instead of sitting on the decode critical path. The replay kernel uses a small fixed grid to keep its SM footprint low while overlapped. The prefetch is enabled automatically for eligible models (no pipeline parallelism, no speculative decoding) and can be turned off for A/B comparison with SGLANG_DISABLE_HISPARSE_PREFETCH=1.

​
Deployment

HiSparse currently requires PD disaggregation mode and is enabled only on the decode instance.

​
Prefill Instance

Command

​
Decode Instance (with HiSparse)

Command
Note: For DSA models, --kv-cache-dtype defaults to auto, which resolves to fp8_e4m3 on SM100+ (Blackwell) and bfloat16 on older architectures. The DSA decode backend is automatically selected based on KV dtype (bfloat16 → flashmla_sparse, fp8_e4m3 → flashmla_kv), except for GLM DSA models on SM120/SM121 with fp8_e4m3, which use flashinfer_sparse_mla. DSA backend flags apply only to DSA models; DeepSeek V4 uses its own dsv4 attention backend.

​
Benchmark

Command

​
Key Notes

  • The prefill instance does not need --enable-hisparse; it is unaware of HiSparse.
  • On the decode instance, --enable-hisparse and --hisparse-config are required for HiSparse.
  • For DSA models, --kv-cache-dtype bfloat16 uses flashmla_sparse, and --kv-cache-dtype fp8_e4m3 uses flashmla_kv.
  • On SM120/SM121 (e.g. RTX PRO 6000, RTX 5090) with GLM DSA models and --kv-cache-dtype fp8_e4m3, both DSA backends resolve to flashinfer_sparse_mla, which is the only DSA kernel available on that architecture. HiSparse accepts it there; no extra flag is needed.
  • For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the dsv4 attention backend and fp8_e4m3 KV cache by default.
  • host_to_device_ratio should be configured based on the host machine’s available memory. For example:
    • ~1 TB host memory → host_to_device_ratio: 5
    • ~2 TB host memory → host_to_device_ratio: 10

​
Acknowledgments

We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions.