> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Ascend NPU Reference This page covers the hardware concepts, communication libraries, deployment terminology, and common abbreviations you will encounter throughout the Ascend NPU documentation. Refer back here when you run into unfamiliar terms. ## Hardware ### Supported devices SGLang supports the following Ascend inference hardware: | Hardware | Chip | Devices | Dies per card | Memory configuration | | ------------------ | ----------- | ------- | ------------- | ------------------------------------ | | Atlas 800I A2 (A2) | Ascend 910B | 8 | 1 | 8(cards) × 1(die/card) × 64(GB/die) | | Atlas 800I A3 (A3) | Ascend 910C | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) | Throughout these docs, **A2** and **A3** are used as shorthand for the hardware above. Docker image tags use `910b` for A2 and `a3` for A3. For example, `v0.5.13.post1-cann9.0.0-910b` and `v0.5.13.post1-cann9.0.0-a3`. On A3, each card has 2 dies, giving 16 devices vs 8 on A2. Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on A3 means `--tp-size 8`. From a deployment perspective, the two key differences between A2 and A3 are: 1. dies per card — which drives both total memory and `--tp-size` configuration 2. PD disaggregation — A2 requires setting `export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"`, while A3 uses the default protocol. ### NPU **NPU** stands for Neural Processing Unit. Each NPU device is a single `davinci` core. The terms "NPU" and "davinci" are used interchangeably in commands and error logs. On A2, devices are numbered `/dev/davinci0` through `/dev/davinci7` (8 devices). On A3, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices). On either an A2 or A3 server, run `npu-smi info` to view NPU information such as device health, memory usage, and chip status. If the command is not found or reports no devices, the driver is likely not installed. Follow the [Ascend driver installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0005.html?OS=openEuler\&InstallType=local) to install it. ## Communication libraries | Library | Description | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **HCCL** (Huawei Collective Communication Library) | The primary communication backend for multi-card data transfer on Ascend NPUs. Equivalent to NVIDIA NCCL. Used in `--nnodes`, `--tp-size`, and all distributed scenarios. | | **GLOO** | Meta's collective communications library. Used alongside HCCL for distributed initialization and coordination. | | **DeepEP** (Deep Expert Parallelism) | A communication library optimized for Mixture-of-Experts (MoE) all-to-all dispatch and combine operations. Used with `--moe-a2a-backend deepep`. | | **RDMA** (Remote Direct Memory Access) | Enables direct memory access between nodes over InfiniBand or RoCE networks. Required for multi-node PD disaggregation. | ## Quantization and precision | Notation | Meaning | | --------- | ------------------------------------------------------------------ | | **W8A8** | 8-bit weights, 8-bit activations | | **W4A8** | 4-bit weights, 8-bit activations | | **W4A16** | 4-bit weights, 16-bit activations | | **BF16** | Brain Floating Point 16 — 16-bit format optimized for ML workloads | | **FP8** | 8-bit Floating Point — not supported on A2/A3 | | **INT8** | 8-bit Integer quantization | To apply quantization, use `--quantization modelslim` for W8A8 INT8, or load a pre-quantized checkpoint directly from a model hub. ## Deployment terminology ### Prefill-Decode (PD) disaggregation PD disaggregation separates inference into two stages running on different nodes: * **Prefill** (P): Processes the entire input prompt at once. Compute-bound. * **Decode** (D): Generates tokens one at a time. Memory-bandwidth-bound. **PD Mixed**: Both stages run on the same set of nodes. | Shorthand | Meaning | | --------- | ------------------------------- | | **1P1D** | 1 prefill node + 1 decode node | | **2P1D** | 2 prefill nodes + 1 decode node | | **1P2D** | 1 prefill node + 2 decode nodes | You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) section headings, e.g. `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes + 1 decode node, 32 cards total." ### Parallelism strategies | Strategy | Flag | Description | | ----------------------------- | ---------------- | ----------------------------------------------------------------------- | | **Tensor Parallelism (TP)** | `--tp-size` | Splits model weights across NPUs within a node | | **Data Parallelism (DP)** | `--dp-size` | Replicates the model across nodes for higher throughput | | **Expert Parallelism (EP)** | `--ep-size` | Distributes MoE experts across devices; requires `--moe-a2a-backend` | | **Context Parallelism (CP)** | `--attn-cp-size` | Splits long context windows across devices for extended sequence length | | **Pipeline Parallelism (PP)** | `--pp-size` | Splits model layers across devices sequentially | ### Speculative decoding | Algorithm | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | **EAGLE3** | Uses an external draft model (specified via `--speculative-draft-model-path`) to propose candidate tokens. Supports top-k sampling. | | **NEXTN** | Uses the model's built-in Multi-Token Prediction (MTP) heads — no separate draft model needed. Available for models with native MTP support. | | **MTP** (Multi-Token Prediction) | A model architecture feature where the model predicts multiple future tokens per step. The foundation for NEXTN speculative decoding. | ### Performance metrics | Metric | Description | | -------------------------------- | ------------------------------------------------------------------------------------ | | **TPOT** (Time Per Output Token) | Average time to generate each output token. Lower is better. | | **TTFT** (Time To First Token) | Latency from request arrival to first generated token. Critical for interactive use. | ## Model architecture terms | Term | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **MoE** (Mixture of Experts) | Model architecture where only a subset of parameters (experts) is activated per token, reducing compute. Common in DeepSeek, Qwen3-30B-A3B, and MiMo models. | | **MLA** (Multi-head Latent Attention) | Attention variant that compresses key-value representations into a lower-dimensional latent space. Used by DeepSeek models. | | **GQA** (Grouped-Query Attention) | Attention variant where multiple query heads share a single key-value head. Used by Qwen3 dense models. | | **DSA** (DeepSeek Sparse Attention) | DeepSeek's sparse attention mechanism; reduces KV cache overhead for long contexts. | | **FFN** (Feed-Forward Network) | The non-attention component of each transformer layer. In MoE models, the FFN is replaced by multiple expert FFN layers selected by a router. | ## Other common abbreviations | Abbreviation | Expansion | | ------------------------------------------------ | ----------------------------------------------------------------------- | | **ACL** (Ascend Computing Language) | Low-level Ascend compute API; seen in error logs | | **DVFS** (Dynamic Voltage and Frequency Scaling) | Hardware frequency scaling to maintain performance stability | | **NUMA** (Non-Uniform Memory Access) | Memory architecture affecting multi-socket performance | | **KV Cache** (Key-Value Cache) | Cached attention key-value tensors to avoid recomputation during decode | | **LoRA** (Low-Rank Adaptation) | Parameter-efficient fine-tuning method | | **HF** (Hugging Face) | Model hub; `HF_TOKEN` / `HF_ENDPOINT` control model download access | | **UB** (Unified Buffer) | On-chip NPU memory; referenced in operator optimization | | **DMA** (Direct Memory Access) | Data transfer mechanism between host and device memory | ## Where to go next * [Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — launch your first server * [Installation Guide](/docs/hardware-platforms/ascend-npus/ascend_npu) — full installation with component version mapping * [Supported Features](/docs/hardware-platforms/ascend-npus/ascend_npu_support_features) — per-parameter Ascend support status * [Supported Models](/docs/hardware-platforms/ascend-npus/ascend_npu_support_models) — models verified on Ascend NPU * [Model Tutorials](/docs/hardware-platforms/ascend-npus/model-tutorials) — step-by-step deployment guides * [Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) — benchmark configurations and results