Installation - SGLang Documentation

You can install SGLang using any of the methods below. Please go through System Settings section to ensure the clusters are operating at optimal performance. Feel free to leave an issue here at sglang if you encounter any issues or have any problems.

Component Version Mapping For SGLang

Each CANN version requires its own matched set of component versions — do not mix versions across sets.

A3 Series
950PR/DT Series

Component	Version	Obtain Method
HDK	25.5.2	HDK driver download
CANN	9.1.0	Obtain Images
TorchNPU	2.10.0.post6	TorchNPU releases
MemFabric	1.2.1	memfabric_hybrid-1.2.1 (x86_64 host: use the manylinux_2_27_x86_64.manylinux_2_28_x86_64 wheel from the same directory)
memcache	1.2.1	memcache_hybrid-1.2.1 (x86_64 host: use the manylinux_2_27_x86_64.manylinux_2_28_x86_64 wheel from the same directory)
Triton	3.2.2	triton_ascend-3.2.2 (x86_64 host: replace aarch64 with x86_64)
SGLang NPU Kernel	2026.9.0.post9	sgl-kernel-npu releases
MemFabric-zbal	1.1.3	`pip install memfabric-zbal==1.1.3`

Component	Version	Obtain Method
HDK	25.5.2	HDK driver download
CANN	9.1.0	Obtain Images
TorchNPU	2.10.0.post6	TorchNPU releases
MemFabric	1.2.1	memfabric_hybrid-1.2.1 (x86_64 host: use the manylinux_2_27_x86_64.manylinux_2_28_x86_64 wheel from the same directory)
memcache	1.2.1	memcache_hybrid-1.2.1 (x86_64 host: use the manylinux_2_27_x86_64.manylinux_2_28_x86_64 wheel from the same directory)
Triton	3.2.2	triton_ascend-3.2.2 (x86_64 host: replace aarch64 with x86_64)
SGLang NPU Kernel	2026.9.0.post9	sgl-kernel-npu releases

Obtain CANN Image

Ensure sufficient disk space before pulling images. Each Docker image requires at least 30GB of free space.

You can obtain the dependency of a specified version of CANN through an image.

A3 Series
950PR/DT Series

Command

docker pull quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12

Command

docker pull quay.io/ascend/cann:9.1.0-950-ubuntu22.04-py3.12

Preparing the Running Environment

Pick one of the two methods below based on your situation:

Method	Recommended scenarios
Using Docker Image	New to SGLang on NPU, or deploying a serving environment
Installing from source with prerequisites	Developing or debugging SGLang, or pinning/customizing component versions

For most users, the Docker image is the recommended path. It is the most convenient option: the matched component set from the version mapping table comes pre-installed, and validated image builds are tested as a whole. Installing from source is the more flexible option: you choose every component version and control the build yourself, with no prebuilt binaries you cannot inspect. Whichever path you take, strictly follow the version mapping table and do not mix versions across CANN sets.

Method 1: Installing from source with prerequisites

The steps in this method install the component set from the version mapping table.

Python Version

The CANN 9.1.0 set requires python==3.12. If you don’t want to break system pre-installed python, try installing with conda.

Command

conda create --name sglang_npu python=3.12
conda activate sglang_npu

Note on Anaconda repository restrictions If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror):

Command

# Add Tsinghua mirrors
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --set show_channel_urls yes
conda config --remove channels defaults

Edit the system-level conda config to remove any hardcoded defaults, e.g., vi ~/miniconda3/.condarc. Then remove the failed environment and recreate it:

Command

conda clean -i
conda env remove -n sglang_npu
conda create --name sglang_npu python=3.12
conda activate sglang_npu

CANN

Prior to starting work with SGLang on Ascend, you need to install CANN Toolkit, Kernels operator package and NNAL version 9.1.0, check the installation guide

MemFabric-Hybrid

If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.

Command

# On an x86_64 host, use memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl from the same directory instead
pip install https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl

# Install the device-specific MemFabric kernels:
# --soc-version is A5 on the 950PR/DT Series, A3 on the A3 Series
mfcli kernel install --soc-version A5

MemFabric-zbal

MemFabric-zbal is a Zero Buffer Acceleration Library of high-performance operators for LLM inference and training on Ascend, accelerating computation by eliminating intermediate memory buffers; it is required only on aarch64 clusters and is installed in addition to MemFabric-Hybrid.

Command

# Only needed on aarch64 (arm64) hosts
# A3 Series:
pip install memfabric-zbal==1.1.3

PyTorch and PyTorch Framework Adapter on Ascend

Command

PYTORCH_VERSION=2.10.0
TORCHVISION_VERSION=0.25.0
TORCH_NPU_VERSION=2.10.0.post6
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION --extra-index-url https://ascend.devcloud.huaweicloud.com/pypi/simple/

If you are using other versions of torch and install torch_npu, check installation guide After the installation, verify that PyTorch can compute on the NPUs — it should print True, then a small tensor computed on the device:

Command

python3 -c "import torch; print(torch.npu.is_available()); x = torch.arange(1, 5, dtype=torch.float32).npu(); print(x * x)"

The expected second line is tensor([1., 4., 9., 16.], device='npu:0') — the squares of 1, 2, 3, 4, computed on the device. You may also see some compatibility warnings caused by device differences; they are safe to ignore.

Triton on Ascend

We provide our own implementation of Triton for Ascend.

Command

# On an x86_64 host, replace aarch64 with x86_64 in the wheel URL
pip install https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl

For installation of Triton on Ascend nightly builds or from sources, follow installation guide

SGLang Kernels NPU

We provide SGL kernels for Ascend NPU, check installation guide.

Gemma-family models (including Qwen3.5) call sgl_kernel_npu.norm.gemma_rmsnorm, whose implementation is chosen when the wheel is built: the native torch_npu operator for A2/A3 series, and ACLNN RMSNorm for 950PR/DT series, which does not register that operator. 950PR/DT series must install the 950 release package (950 appears in the asset name), or build from source with bash build.sh -a kernels Ascend950PR_9599, the target that package is built with. On 950PR/DT, an a2 or a3 package fails on the first Gemma forward, and so does a package from a release that predates this module; check with python -c "import sgl_kernel_npu.norm.gemma_rmsnorm".

DeepEP-compatible Library

We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai’s DeepEP library, check the installation guide. It is optional and needed only when serving MoE models with expert parallelism through --moe-a2a-backend deepep; plain TP deployments can skip it.

Additional dependencies

Command

# libGL
apt update
apt install libgl1 libglib2.0-0

# ensure setuptools contains pkg_resources module
pip install "setuptools<80"

Installing SGLang from source

Command

# Use the last release branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
mv python/pyproject_npu.toml python/pyproject.toml
pip install -e python[all_npu]

Method 2: Using Docker Image

Obtain Image

You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.

Ensure sufficient disk space before pulling images. Each Docker image requires at least 30GB of free space. If you need to download model weights, check the model size at ModelScope to reserve enough space.

Download SGLang image

We publish both stable releases and daily builds. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., main-cann9.1.0-a3, main-cann9.1.0-950) if you need the latest development changes.CANN 9.1.0 has no stable release yet, so use a validated build for now. The cann9.1.0-<device>-B<NN> tags (e.g., cann9.1.0-a3-B105, cann9.1.0-950-B105) are validated builds rather than stable releases; they stand in until a stable release is published. The exact component versions inside each build are listed in the image dependency manifest — see for example the cann9.1.0-950-B105 entry for the 950PR/DT Series.

A3 Series
950PR/DT Series

Command

docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-B105

Command

docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B105

Build an image based on Dockerfile

Command

# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker

# Build the docker image
# Replace <arch_tag> with the target architecture, e.g., amd64, arm64.
# Optional build arguments:
#   --build-arg APTMIRROR=<mirror_url>    # Use a custom APT mirror to improve download speed
# If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV.
docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile .

Create Docker Container

Notice: --privileged and --network=host are required by RDMA, which is typically needed by Ascend NPU clusters. Model weights: set HF_TOKEN to download automatically, or replace the model ID in --model-path with a local weights path — see How do I provide model weights? in the FAQ.

A3 Series
950PR/DT Series

Command

# Create a shortcut 'drun' to launch a privileged Docker container
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'

# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
drun --env "HF_TOKEN=<secret>" \
    <image_name> \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend

Command

# Create a shortcut 'drun' to launch a privileged Docker container
# The 950PR/DT Series exposes 8 devices: /dev/davinci0 through /dev/davinci7
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --env ASCEND_USE_FIA=1 \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'

# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
drun --env "HF_TOKEN=<secret>" \
    <image_name> \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend

SGLang will serve on http://127.0.0.1:30000 by default. You can change the host and port by --host and --port parameters.

System Settings

CPU performance power scheme

The default power scheme on Ascend hardware is ondemand which could affect performance, changing it to performance is recommended.

Command

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance

Disable NUMA balancing

The kernel’s background NUMA page migration adds memory-bandwidth overhead and latency jitter to SGLang’s large resident working set (weights and KV cache); disabling it keeps serving latency stable.

Command

sudo sysctl -w kernel.numa_balancing=0
# Check
cat /proc/sys/kernel/numa_balancing # shows 0

Prevent swapping out system memory

Command

sudo sysctl -w vm.swappiness=10

# Check
cat /proc/sys/vm/swappiness # shows 10

Running SGLang Service

Running Service For Large Language Models

Pick a deployment mode based on your workload:

Mode	Characteristics	Recommended scenarios
PD mixed	Prefill and decode in one server: single process, no router, requests go straight to the server; heavy prefill batches share hardware with decode and can raise inter-token latency	Getting started, small to medium scale, no strict inter-token-latency requirement
PD disaggregation	Dedicated prefill and decode pools behind a router, KV cache transferred over the network (requires MemFabric-Hybrid); each phase scales and tunes independently, at the cost of more components and nodes	Large scale, long prompts, high concurrency, where stable inter-token latency matters

PD Mixed Scene

Model weights: set HF_TOKEN to download automatically, or replace the model ID in --model-path with a local weights path — see How do I provide model weights? in the FAQ.

A3 Series
950PR/DT Series

Command

# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --attention-backend ascend \
  --host 127.0.0.1 \
  --port 30000

Command

# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --attention-backend ascend \
  --host 127.0.0.1 \
  --port 30000

The example above uses meta-llama/Llama-3.1-8B-Instruct. To run another model, change the --model-path value and make sure it is listed in Supported Models.

PD Disaggregation Scene

PD disaggregation requires the prefill server, the decode server, and the router to communicate with one another. Make sure the hosts can reach one another over the network and that the PREFILL_IP:FREE_PORT store address is accessible from every node. The example below runs all three on a single machine, each using its own port and NPU (via --base-gpu-id); the same steps apply to a multi-node cluster, provided the hosts can reach one another.

Launch Prefill Server

Model weights: set HF_TOKEN to download automatically, or replace the model ID in --model-path with a local weights path — see How do I provide model weights? in the FAQ.

A3 Series
950PR/DT Series

Command

# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30001

Command

# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
# PD disaggregation transfer protocol on the 950PR/DT Series
export ASCEND_MF_TRANSFER_PROTOCOL="device_urma"

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30001

Launch Decode Server

Model weights: set HF_TOKEN to download automatically, or replace the model ID in --model-path with a local weights path — see How do I provide model weights? in the FAQ.

A3 Series
950PR/DT Series

Command

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 1 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30002

Command

# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
# PD disaggregation transfer protocol on the 950PR/DT Series
export ASCEND_MF_TRANSFER_PROTOCOL="device_urma"

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 1 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30002

Launch Router

The router command requires the sglang-router package. It is pre-installed in the Docker image; if you installed SGLang from source and do not have it yet, run pip install sglang-router first.

Command

python3 -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://127.0.0.1:30001 8995 \
    --decode http://127.0.0.1:30002 \
    --host 127.0.0.1 \
    --port 30000

The 8995 in command script is the disaggregation bootstrap port. It must match the --disaggregation-bootstrap-port value set on the prefill server in step 1.

Running Service For Multimodal Language Models

PD Mixed Scene

Model weights: set HF_TOKEN to download automatically, or replace the model ID in --model-path with a local weights path — see How do I provide model weights? in the FAQ.

A3 Series
950PR/DT Series

Command

python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
    --host 127.0.0.1 \
    --port 30000 \
    --tp 4 \
    --device npu \
    --attention-backend ascend \
    --mm-attention-backend ascend_attn \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-multimodal \
    --sampling-backend ascend

Command

# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
    --host 127.0.0.1 \
    --port 30000 \
    --tp 4 \
    --device npu \
    --attention-backend ascend \
    --mm-attention-backend ascend_attn \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-multimodal \
    --sampling-backend ascend

The example above uses Qwen/Qwen3-VL-30B-A3B-Instruct. For other multimodal models, see the multimodal section of Supported Models.

Testing the Service

Once the server prints The server is fired up and ready to roll! in the logs, it is ready to accept requests.

Which port to send requests to

The port you use depends on your deployment mode:

Scenario	Where to send requests
Non-PD (single server)	The server’s `--port` (e.g., `30000` in the examples above)
Non-PD (multi-node)	The primary node’s (`--node-rank 0`) `--port`; do not send requests to worker nodes
PD disaggregation	The router’s `--port` (e.g., `30000` in the examples above); do not send requests directly to prefill or decode servers

SGLang defaults to port 30000 when --port is not specified. The examples in this guide use explicit ports for clarity.

Health Check

Command

curl http://127.0.0.1:30000/health

A successful response returns HTTP 200 with an empty body.

Generate (Native Endpoint)

Command

curl http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "What is the capital of France?",
    "sampling_params": {"temperature": 0, "max_new_tokens": 128}
  }'

The expected output should contain “Paris”.

Chat Completions (OpenAI-Compatible)

Command

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

Some models return responses accompanied by thinking process content. To disable this output, configure parameters as follows:

Command

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The expected output should contain “Paris”.

Multimodal Chat Completions

The image URL in the example below references an external resource (raw.githubusercontent.com). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data.

Command

curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-VL-30B-A3B-Instruct",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
        {"type": "text", "text": "Describe this image."}
      ]
    }]
  }'

FAQ

How do I provide model weights?

The launch commands in this guide pass a Hugging Face model ID (e.g., meta-llama/Llama-3.1-8B-Instruct) to --model-path. Pick one way to provide the weights:

Automatic download: export HF_TOKEN (required for gated models such as Llama) and let SGLang download the weights from Hugging Face on first launch; they are cached under ~/.cache/huggingface and reused afterwards.
Pre-downloaded weights: download the weights to the machine yourself, then replace the model ID in the commands with the local weights path, e.g., --model-path /path/to/Llama-3.1-8B-Instruct.

When running in Docker, the drun alias mounts ~/.cache/ from the host, so weights already in the host’s Hugging Face cache are visible inside the container.

Next Steps

For mainstream models, we provide step-by-step deployment tutorials and tuned, best-practice launch scripts — see Model Tutorials and Best Practices.

​Component Version Mapping For SGLang

​Obtain CANN Image

​Preparing the Running Environment

​Method 1: Installing from source with prerequisites

​Python Version

​CANN

​MemFabric-Hybrid

​MemFabric-zbal

​PyTorch and PyTorch Framework Adapter on Ascend

​Triton on Ascend

​SGLang Kernels NPU

​DeepEP-compatible Library

​Additional dependencies

​Installing SGLang from source

​Method 2: Using Docker Image

​Obtain Image

​Create Docker Container

​System Settings

​CPU performance power scheme

​Disable NUMA balancing

​Prevent swapping out system memory

​Running SGLang Service

​Running Service For Large Language Models

​PD Mixed Scene

​PD Disaggregation Scene

​Running Service For Multimodal Language Models

​PD Mixed Scene

​Testing the Service

​Which port to send requests to

​Health Check

​Generate (Native Endpoint)

​Chat Completions (OpenAI-Compatible)

​Multimodal Chat Completions

​FAQ

​How do I provide model weights?

​Next Steps

Component Version Mapping For SGLang

Obtain CANN Image

Preparing the Running Environment

Method 1: Installing from source with prerequisites

Python Version

CANN

MemFabric-Hybrid

MemFabric-zbal

PyTorch and PyTorch Framework Adapter on Ascend

Triton on Ascend

SGLang Kernels NPU

DeepEP-compatible Library

Additional dependencies

Installing SGLang from source

Method 2: Using Docker Image

Obtain Image

Create Docker Container

System Settings

CPU performance power scheme

Disable NUMA balancing

Prevent swapping out system memory

Running SGLang Service

Running Service For Large Language Models

PD Mixed Scene

PD Disaggregation Scene

Running Service For Multimodal Language Models

PD Mixed Scene

Testing the Service

Which port to send requests to

Health Check

Generate (Native Endpoint)

Chat Completions (OpenAI-Compatible)

Multimodal Chat Completions

FAQ

How do I provide model weights?

Next Steps