Skip to main content
Decode context parallelism (DCP) stripes a request’s MLA KV cache across ranks in an existing tensor-parallel group. With DCP size c, rank r owns position p when p mod c = r, so each rank stores and reads roughly 1/c of the cache. Local attention then sees only part of the context. The MLA path returns a partial output and log-sum-exp (LSE), exchanges both in one packed all-to-all, and merges them exactly. DCP complements TP and attention data parallelism (DPA) rather than replacing them. The primary path is absorbed MLA decode and static target verification. dcp_size=1 retains existing non-DCP behavior. On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchanged, so DCP grows long-context KV capacity but does not raise the KDA concurrency ceiling.

​
Enable DCP

A DCP group must fit inside one attention-TP group and one attention-DP replica:
Topology
For example, TP=64, DP=4, DCP=16 creates four valid 16-rank attention replicas. DCP=32 would cross replica boundaries and is invalid.
Startup currently checks only tp_size % dcp_size == 0, not the stronger condition above. Until that check is fixed, DPA+DCP deployments must enforce containment when choosing the topology.
MLA decode on DeepSeek-V3.1 with DCP 8 inside TP 8:
Command
Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default and picks fi_a2a on MNNVL systems, otherwise a2a:
Command
See the Kimi K3 cookbook for large-scale presets that combine DCP with DPA and expert parallelism.

​
Why decode context parallelism?

Absorbed MLA shares one latent KV representation across all query heads. TP can split the heads and weights, but not that cache, so every attention-TP rank stores and reads the same context. DCP stripes that cache round-robin by logical token position, keeps a shared virtual index on every rank, runs attention against the owner-local shard, and merges partial outputs by LSE into the usual TP-local head layout.

​
Architecture

​
Virtual KV layout

Let each rank have physical token capacity C and physical page size P. DCP exposes a shared virtual allocator:
Layout
Every rank sees the same request-to-token map. On write, it keeps its positions and compacts them into the local physical pool. One widened virtual page maps to one physical page on each rank, so the layout stays balanced to within one token as a sequence grows.

​
MLA decode dataflow

For context partition r, the kernel returns locally normalized output o_r and lse_r. The global result is lse = logsumexp_r(lse_r) and o = sum_r(exp(lse_r - lse) * o_r). This is exact aside from floating-point reduction order. The merge uses the attention backend’s LSE base, either base-e or base-2.

​
Query-projection replication

Normally each layer all-gathers its absorbed and rotary query components. --dcp-replicate-q-proj gathers the query-projection and w_kc weights once at startup and computes the full DCP-group query locally. The trade is more weight memory and GEMM work for one fewer collective per layer. Only unquantized BF16/FP16 weights use this path; other layers fall back to the query all-gather.

​
Communication backends

Kimi K3 enables replicated Q by default and picks fi_a2a on MNNVL systems, otherwise a2a. Its decode backend is cutedsl_mla. The generic defaults remain ag_rs and model-resolved query replication. DCP adds a context-independent decode collective while cutting context-dependent KV storage and reads by about c. Extend is outside that decode cost model: it may gather cached prefix shards, restore token order, and append new tokens, and that work grows with prefix length.

​
Compositions

​
DCP × speculative decoding

The draft KV cache is replicated: every DCP rank stores the full draft context. DCP only stripes the target MLA KV, so it saves target cache, not draft cache. Target verify and decode use the DCP-aware MLA kernel cutedsl_mla. It:
  1. Builds a rank-local page table from the cyclic token map (owner(p) = p mod c).
  2. Runs attention against that shard, passing cp_world, cp_rank, and the global KV lengths so causality still sees the full sequence.
  3. Returns a rank-local (output, LSE) that the usual DCP all-to-all merge combines.
Draft steps use the same backend against the replicated draft cache, so they skip DCP metadata. Kimi K3 DCP selects cutedsl_mla automatically.

​
DCP × PD disaggregation

A dense prefill page cannot be copied directly into striped decode storage. The PD sender uses the decode rank’s DCP metadata to select and pack the right rows. See PD disaggregation for the transfer engines.
  • Equal DCP sizes require matching DCP ranks and use the existing page path.
  • DCP1 prefill to DCP decode uses token relayout for MLA and hybrid-MLA pools.
  • All other DCP-size transitions are rejected.
The plan includes the exact token count, so stale rows do not leak out of a partial final page. Mooncake and NIXL use the same plan. KDA state keeps its attention-TP mapping and bypasses DCP filtering. This composition requires Mooncake or NIXL, matching physical page size and KV dtype, prefill attention CP 1, and decode chunk cache. Decode radix cache and HiCache are not supported here.

​
DCP × HiCache L2

HiCache keeps one widened logical page space across L1 and L2. The controller sees P * c slots, while each GPU and host buffer stores only its P local rows. See HiCache for the cache hierarchy. Before H2D or D2H, both index lists keep positions owned by this rank and map i to floor(i / c). Transfers cover whole widened pages, and every rank moves its shard independently, with no DCP collective. For Kimi K3 this applies to MLA KV; KDA/Mamba state uses its own request-indexed host pool. The current scope is MLA L1/L2. L3, LMCache, HiSparse, non-MLA KV host pools, speculative decoding, and PD decode are not supported in this combination.

​
References