MiniMax-M2.5 - SGLang Documentation

This guide describes the best practice data for MiniMax-M2.5 on the Ascend NPU.

Low Latency

Model	Hardware	Cards	Deploy Mode	Dataset	TPOT	Quantization	Configuration
MiniMax-M2.5	Atlas 800I A3	8	PD Mixed	128K+1K (90% prefix cache hit rate)	20ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	8	PD Mixed	3.5K+1.5K	20ms	W8A8 INT8	Optimal Configuration

High Throughput

Model	Hardware	Cards	Deploy Mode	Dataset	TPOT	Quantization	Configuration
MiniMax-M2.5	Atlas 800I A3	16	PD Disaggregation	128K+1K (90% prefix cache hit rate)	50ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	16	PD Disaggregation	64K+1K (90% prefix cache hit rate)	50ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	4	PD Mixed	32K+1K	50ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	4	PD Mixed	64K+1K (90% prefix cache hit rate)	50ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	8	PD Mixed	128K+1K (90% prefix cache hit rate)	50ms	W8A8 INT8	Optimal Configuration
MiniMax-M2.5	Atlas 800I A3	8	PD Mixed	3.5K+1.5K	50ms	W8A8 INT8	Optimal Configuration

Optimal Configuration

MiniMax-M2.5 W8A8 1P1D 16P IN128K OUT1K PREFIX90 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 16 Deploy Mode: PD Disaggregation Quantization: W8A8 INT8 Dataset: 128K+1K (90% prefix cache hit rate) TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export ASCEND_USE_FIA=1
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=64
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export ENABLE_PROFILING=0
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=2500
        export HCCL_SOCKET_IFNAME=<network-interface>
        export PROFILING_BS=8
        export PROFILING_STAGE=prefill
        export PROFILING_step=30
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --trust-remote-code \
        --tp-size 16 \
        --mem-fraction-static 0.43 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 128 \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 130000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --tokenizer-worker-num 16 \
        --dp-size 2 \
        --enable-dp-attention \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 2 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 3 \
        --speculative-draft-model-quantization unquant \
        --skip-server-warmup
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1600
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640
        export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=96
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_NPU_FUSED_MOE_MODE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --trust-remote-code \
        --tp-size 16 \
        --mem-fraction-static 0.76 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 80 \
        --chunked-prefill-size -1 \
        --moe-a2a-backend ascend_fuseep \
        --deepep-mode low_latency \
        --tokenizer-worker-num 8 \
        --dp-size 2 \
        --enable-dp-attention \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 2 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 3 \
        --speculative-draft-model-quantization unquant \
        --skip-server-warmup \
        --cuda-graph-bs 2 4 8
        NODE_RANK=$i
        break
    fi
done

Command

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin --mini-lb

Benchmark

We tested it based on the generated-shared-prefix dataset with 90% cache hit (repeat_rate = 0.9): --gsp-system-prompt-len 117964 = int(131072 * 0.9) is the shared prefix portion. --gsp-question-len 13107 = int(131072 * (1 - 0.9)) is the unique per-request suffix. --gsp-num-groups 1 keeps all requests in one prefix group for maximum cache reuse.

Command

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 32 \
    --gsp-system-prompt-len 117964 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 8 \
    --num-prompts 32 \
    --request-rate inf

MiniMax-M2.5 W8A8 1P1D 16P IN64K OUT1K PREFIX90 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 16 Deploy Mode: PD Disaggregation Quantization: W8A8 INT8 Dataset: 64K+1K (90% prefix cache hit rate) TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export ASCEND_USE_FIA=1
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=64
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export ENABLE_PROFILING=0
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=2500
        export HCCL_SOCKET_IFNAME=<network-interface>
        export PROFILING_BS=8
        export PROFILING_STAGE=prefill
        export PROFILING_step=30
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --trust-remote-code \
        --tp-size 16 \
        --mem-fraction-static 0.43 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 128 \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 58000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --tokenizer-worker-num 16 \
        --dp-size 2 \
        --enable-dp-attention \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --skip-server-warmup
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1600
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640
        export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=96
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_NPU_FUSED_MOE_MODE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --trust-remote-code \
        --tp-size 16 \
        --mem-fraction-static 0.76 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 80 \
        --chunked-prefill-size -1 \
        --moe-a2a-backend ascend_fuseep \
        --deepep-mode low_latency \
        --tokenizer-worker-num 16 \
        --dp-size 2 \
        --enable-dp-attention \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --skip-server-warmup \
        --cuda-graph-bs 2 4 8 16 24 32 40
        NODE_RANK=$i
        break
    fi
done

Command

# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin --mini-lb

Benchmark

We tested it based on the generated-shared-prefix dataset with 90% cache hit (repeat_rate = 0.9): --gsp-system-prompt-len 58982 = int(65536 * 0.9) is the shared prefix portion. --gsp-question-len 6553 = int(65536 * (1 - 0.9)) is the unique per-request suffix. --gsp-num-groups 1 keeps all requests in one prefix group for maximum cache reuse.

Command

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 640 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6553 \
    --gsp-output-len 1024 \
    --max-concurrency 160 \
    --num-prompts 640 \
    --request-rate inf

MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 4 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 32K+1K TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=64
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=128
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_LOCAL_MEM_SIZE=60184
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBAL_ENABLE_GRAPH=1
export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 8 \
    --disable-radix-cache \
    --mem-fraction-static 0.74 \
    --max-running-requests 18 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 32768 \
    --cuda-graph-bs 2 4 6 8 10 12 14 16 18 24 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --trust-remote-code \
    --tokenizer-worker-num 4

Benchmark

We tested it based on the RANDOM dataset.

Command

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 18 \
    --random-input-len 32768 \
    --random-output-len 1024 \
    --num-prompts 72 \
    --random-range-ratio 1

MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 4 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 64K+1K (90% prefix cache hit rate) TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1024
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=140000
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 8 \
    --mem-fraction-static 0.63 \
    --max-running-requests 26 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --enable-prefill-delayer \
    --prefill-max-requests 10 \
    --chunked-prefill-size 67072 \
    --max-prefill-token 67000 \
    --cuda-graph-bs 2 4 8 12 16 18 20 22 24 26 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --trust-remote-code

Benchmark

Command

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 104 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6553 \
    --gsp-output-len 1024 \
    --max-concurrency 26 \
    --num-prompts 104 \
    --request-rate inf

MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 20ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 8 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 128K+1K (90% prefix cache hit rate) TPOT: 20ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1024
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --dp-size 2 \
    --enable-dp-attention \
    --mem-fraction-static 0.65 \
    --max-running-requests 4 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --enable-prefill-delayer \
    --prefill-max-requests 4 \
    --chunked-prefill-size 160000 \
    --max-prefill-token 80000 \
    --cuda-graph-bs 2 4 6 8 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --tokenizer-worker-num 4 \
    --dtype bfloat16

Benchmark

Command

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 16 \
    --gsp-system-prompt-len 117964 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 4 \
    --num-prompts 16 \
    --request-rate inf

MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 8 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 128K+1K (90% prefix cache hit rate) TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1024
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --dp-size 2 \
    --enable-dp-attention \
    --mem-fraction-static 0.65 \
    --max-running-requests 20 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --enable-prefill-delayer \
    --prefill-max-requests 4 \
    --chunked-prefill-size 160000 \
    --max-prefill-tokens 80000 \
    --cuda-graph-bs 2 4 6 8 10 16 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --tokenizer-worker-num 4 \
    --dtype bfloat16

Benchmark

Command

python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 80 \
    --gsp-system-prompt-len 117964 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 20 \
    --num-prompts 80 \
    --request-rate inf

MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 8 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 3.5K+1.5K TPOT: 20ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=2048
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --enable-dp-attention \
    --dp-size 16 \
    --mem-fraction-static 0.53 \
    --max-running-requests 96 \
    --disable-radix-cache \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --prefill-delayer-max-delay-passes 500 \
    --enable-prefill-delayer \
    --prefill-max-requests 3 \
    --chunked-prefill-size -1 \
    --max-prefill-token 8192 \
    --cuda-graph-bs 1 2 3 4 5 6 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16

Benchmark

We tested it based on the RANDOM dataset.

Command

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 112 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --num-prompts 448 \
    --random-range-ratio 1

MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms

Model: MiniMax-M2.5 Hardware: Atlas 800I A3 Cards: 8 Deploy Mode: PD Mixed Quantization: W8A8 INT8 Dataset: 3.5K+1.5K TPOT: 50ms

Model Deployment

Command

# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1024
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --enable-dp-attention \
    --dp-size 16 \
    --mem-fraction-static 0.75 \
    --max-running-requests 320 \
    --disable-radix-cache \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --prefill-delayer-max-delay-passes 500 \
    --enable-prefill-delayer \
    --chunked-prefill-size -1 \
    --max-prefill-token 8192 \
    --cuda-graph-bs 1 2 4 8 12 16 20 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16

Benchmark

We tested it based on the RANDOM dataset.

Command

python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 320 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --num-prompts 1280 \
    --random-range-ratio 1

​Low Latency

​High Throughput

​Optimal Configuration

​MiniMax-M2.5 W8A8 1P1D 16P IN128K OUT1K PREFIX90 50ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 1P1D 16P IN64K OUT1K PREFIX90 50ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 20ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 50ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms

​Model Deployment

​Benchmark

​MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms

​Model Deployment

​Benchmark

Low Latency

High Throughput

Optimal Configuration

MiniMax-M2.5 W8A8 1P1D 16P IN128K OUT1K PREFIX90 50ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 1P1D 16P IN64K OUT1K PREFIX90 50ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 20ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 50ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms

Model Deployment

Benchmark

MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms

Model Deployment

Benchmark