Cambricon MLUs - SGLang Documentation

This guide describes how to prepare the Cambricon software environment and install SGLang for MLU CI environment validation. MLU inference support is provided separately.

Prerequisites

Install the MLU driver and firmware on the host before proceeding. Select packages matching the component version configuration, container OS, CPU architecture, and Python version.

MLU software components

Component versions are defined in scripts/ci/mlu/component_versions.properties. Before installing SGLang, install the components at the versions specified in this file for the SGLang commit you intend to run. The table below lists the MLU software components and their roles.

Category	Component	Purpose
PyTorch stack	`pytorch`	CPU build of the PyTorch deep learning framework
	`torchvision`	CPU build of the PyTorch vision package
	`torchaudio`	CPU build of the PyTorch audio package
	`torch_mlu`	PyTorch MLU device adapter
	`torch_mlu_ops`	PyTorch third-party operator library on MLU
Toolkit	`cntoolkit`	Cambricon Neuware Toolkit, distributed as a bundle of development and runtime packages
Operator libraries	`cnnl`	Cambricon Network Library for neural network operators
	`cnnlextra`	CNNL Extra Library, including transformer, quantization, and detection operators
	`mlu-ops`	Cambricon open-source operator library for neural networks
Communication libraries	`cncl`	Cambricon Communication Library for communicator management and collective communication
Communication libraries	`cnclep`	Cambricon Neuware Communication Library for Expert Parallelism, providing high-throughput, low-latency MoE communication
Kernel programming	`triton`	A language and compiler for custom deep learning operations on the MLU backend
Disaggregated architecture	`mooncake`	MLU-adapted Mooncake with a KV cache-centric architecture for disaggregated LLM inference and training
Distributed execution	`ray`	MLU-adapted Ray for distributed computing and scaling AI workloads

Installation

Create a Docker container

Contact Cambricon official support at ecosystem@cambricon.com to obtain the container image. Set MLU_IMAGE to the image address supplied by Cambricon, using the tag specified by MLU_CI_IMAGE_TAG in the version configuration. The example below uses two MLU devices on an x86_64 host and a container with Ubuntu and Python matching that image tag. Run on the host:

docker run -it --user root --network=host --shm-size=16g \
  --device=/dev/cambricon_ctl \
  --device=/dev/cambricon_dev0 --device=/dev/cambricon_dev1 \
  --workdir /workspace \
  "$MLU_IMAGE" bash

The device mappings above use the nodes exposed by the reference host driver. Adjust them to match your host’s device configuration. Run the remaining installation commands inside the container.

Install the MLU components

Contact Cambricon official support at ecosystem@cambricon.com to obtain the required software packages and installation guidance.

Install SGLang

Use the Python environment where you installed the MLU components:

git clone https://github.com/sgl-project/sglang.git
cd sglang

mv python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_mlu]"

MLU installation uses the shared pyproject_other.toml configuration. The all_mlu extra includes the runtime dependencies and tracing support. The native libraries and framework packages must already be installed.

Verify the CI environment

Install the test dependencies, then run the single-device RMSNorm and two-device SGLang communicator suites:

pip install -e "python[dev_mlu]"
python test/run_suite.py --hw mlu --suite pr-test-1-mlu
python test/run_suite.py --hw mlu --suite pr-test-2-mlu

These tests validate the MLU RMSNorm implementation and SGLang communicator collectives without loading a model or starting an inference server.

​Prerequisites

​MLU software components

​Installation

​Create a Docker container

​Install the MLU components

​Install SGLang

​Verify the CI environment