> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sglang.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Qwen3.6-35B-A3B

<Note>
  This page focuses on optimal configuration and benchmark results for Qwen3.6-35B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-35B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b).

  On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware) for details.
</Note>

### Low Latency

| Model           | Hardware      | Cards | Deploy Mode | Dataset | TPOT   | Quantization | Configuration                                             |
| --------------- | ------------- | ----- | ----------- | ------- | ------ | ------------ | --------------------------------------------------------- |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 254k+1k | 16.1ms | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) |

### High Throughput

| Model           | Hardware      | Cards | Deploy Mode | Dataset                            | TPOT    | Quantization | Configuration                                                            |
| --------------- | ------------- | ----- | ----------- | ---------------------------------- | ------- | ------------ | ------------------------------------------------------------------------ |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 1024x1024 (30)+1024                | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 1080p\_30+256                      | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms)      |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 128k+1k                            | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms)           |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 3.5k+1.5k                          | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms)           |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 64k+1k                             | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms)            |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 1     | PD Mixed    | 64k+1k (90% prefix cache hit rate) | 50ms    | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms)   |
| Qwen3.6-35B-A3B | Atlas 800I A3 | 2     | PD Mixed    | 984k+1k                            | 40.91ms | BF16         | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k)                |

## Optimal Configuration

<a id="single-node-pd-mixed" title="Referenced by external docs. Verify before removing." />

### Qwen3.6-35B-A3B 1P IN1024X1024 30 OUT1024 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1024x1024 (30)+1024

*Format: resolution (input tokens) + output tokens*

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 16384 \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-prefill-delayer \
    --max-running-requests 120 \
    --max-mamba-cache-size 240 \
    --mem-fraction-static 0.78 \
    --cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 120 \
    --random-input-len 30 \
    --random-output-len 1024 \
    --num-prompts 480 \
    --random-range-ratio 1 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 1P IN1080P 30 OUT256 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1080p\_30+256

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=10
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 16384 \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-prefill-delayer \
    --max-running-requests 50 \
    --max-mamba-cache-size 55 \
    --mem-fraction-static 0.8 \
    --cuda-graph-bs 2 4 8 12 16 20 24 28 32 36 40 44 48 50 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 50 \
    --random-input-len 30 \
    --random-output-len 256 \
    --num-prompts 200 \
    --random-range-ratio 1 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 1P IN128K OUT1K 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 128k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1600
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 520960 \
    --max-prefill-tokens 128000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 3 \
    --max-mamba-cache-size 10 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs 1 2 3 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 3 \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --num-prompts 3 \
    --random-range-ratio 1
```

### Qwen3.6-35B-A3B 1P IN254K OUT1K

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 254k+1k

**TPOT**: 16.1ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 131072 \
    --max-prefill-tokens 254000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 1 \
    --max-mamba-cache-size 6 \
    --mem-fraction-static 0.65 \
    --cuda-graph-bs 1 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --random-input-len 254000 \
    --random-output-len 1000 \
    --num-prompts 1 \
    --random-range-ratio 1
```

### Qwen3.6-35B-A3B 1P IN3K5 OUT1K5 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 659840 \
    --max-prefill-tokens 43400 \
    --disable-radix-cache \
    --trust-remote-code \
    --prefill-max-requests 12 \
    --max-running-requests 122 \
    --max-mamba-cache-size 122 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs 4 16 32 64 96 116 120 122 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 122 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --num-prompts 122 \
    --random-range-ratio 1
```

### Qwen3.6-35B-A3B 1P IN64K OUT1K 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 600000 \
    --max-prefill-tokens 65536 \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-prefill-delayer \
    --max-running-requests 10 \
    --max-mamba-cache-size 20 \
    --mem-fraction-static 0.65 \
    --cuda-graph-bs 2 4 8 12 14 16 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 10 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --num-prompts 40 \
    --random-range-ratio 1
```

### Qwen3.6-35B-A3B 1P IN64K OUT1K PREFIX90 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 64k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=300
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 470784 \
    --max-prefill-tokens 65536 \
    --trust-remote-code \
    --mamba-radix-cache-strategy extra_buffer \
    --max-running-requests 40 \
    --max-mamba-cache-size 200 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs 2 8 16 24 32 36 40 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 58982` = `int(65536 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6553` = `int(65536 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 40 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6553 \
    --gsp-output-len 1024 \
    --max-concurrency 40 \
    --num-prompts 40 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 2P IN984K OUT1K

**Model**: Qwen3.6-35B-A3B

**Hardware**: Atlas 800I A3

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 984k+1k

**TPOT**: 40.91ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 131072 \
    --max-prefill-tokens 984000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 1 \
    --max-mamba-cache-size 6 \
    --mem-fraction-static 0.68 \
    --cuda-graph-bs 1 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --context-length 1010000 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --random-input-len 984000 \
    --random-output-len 1000 \
    --num-prompts 1 \
    --random-range-ratio 1
```