Skip to main content

1. Model Introduction

Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, representing a significant leap forward with unified vision-language foundation, efficient hybrid architecture, and scalable reinforcement learning. Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated), delivering high-throughput inference with minimal latency. It supports multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens. Architecture details:
  • Hybrid Attention: Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer — linear layers provide low-cost long-context processing while periodic full attention ensures high associative recall.
  • MoE routing: Top-10 active out of 512 routed experts plus a dedicated shared expert for universal features, keeping 17B parameters active from 397B total.
  • Native multimodal: DeepStack Vision Transformer with Conv3d temporal encoding for image and video understanding without separate visual encoders.
Key Features:
  • Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models
  • Efficient Hybrid Architecture: Gated Delta Networks + sparse MoE (397B total / 17B active) for high-throughput inference
  • Hybrid Reasoning: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
  • Tool Calling: Built-in tool calling support with qwen3_coder parser
  • Multi-Token Prediction (MTP): Speculative decoding support for lower latency
  • 201 Language Support: Expanded multilingual coverage across 201 languages and dialects
Available Models:
ModelBF16 (Full precision)FP8 (8-bit Quantized)FP4 (4-bit Quantized)
Qwen3.5-397B-A17BQwen/Qwen3.5-397B-A17BQwen/Qwen3.5-397B-A17B-FP8NVIDIA NVFP4: nvidia/Qwen3.5-397B-A17B-NVFP4-V2
AMD MXFP4: amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2
Qwen3.5-122B-A10BQwen/Qwen3.5-122B-A10BQwen/Qwen3.5-122B-A10B-FP8-
Qwen3.5-35B-A3BQwen/Qwen3.5-35B-A3BQwen/Qwen3.5-35B-A3B-FP8-
Qwen3.5-27BQwen/Qwen3.5-27BQwen/Qwen3.5-27B-FP8-
Qwen3.5-9BQwen/Qwen3.5-9B--
Qwen3.5-4BQwen/Qwen3.5-4B--
Qwen3.5-2BQwen/Qwen3.5-2B--
Qwen3.5-0.8BQwen/Qwen3.5-0.8B--
License: Apache 2.0

2. SGLang Installation

SGLang from the main branch is required for Qwen3.5. You can install from source or use a Docker image:
Command
For the full Docker setup and other installation methods, please refer to the official SGLang installation guide.

3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.

3.1 Basic Configuration

Interactive Command Generator: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.

3.2 Configuration Tips

  • Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
  • H100 / B200 / B300 FP8: Add --enable-symm-mem to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. It only helps when tp > 1, so on B200/B300 it applies to the 397B-A17B FP8 recipe (tp=4); the 122B-A10B and 35B-A3B FP8 recipes are single-GPU there.
  • AMD GPUs (MI300X / MI325X / MI355X): Use SGLANG_USE_AITER=1 and SGLANG_USE_AITER_UNIFIED_ATTN=1 with --attention-backend aiter, which requires --page-size 16 and can also enable --enable-aiter-allreduce-fusion. Additionally set AITER_FLYDSL_FORCE=1 to force the AITER FlyDSL MoE kernels and SGLANG_MAMBA_SSM_DTYPE=bfloat16 to store the Mamba SSM state in bfloat16 (instead of the default float32). For the MXFP4 checkpoint on MI355X, set ROCM_QUICK_REDUCE_QUANTIZATION=INT4 to route multi-GPU collectives through INT4-quantized ROCm quick all-reduce, and drop --enable-aiter-allreduce-fusion (the two are mutually exclusive; quick all-reduce is preferred for this recipe). If the MXFP4 checkpoint keeps its shared expert unquantized, --enforce-shared-experts-fusion quantizes it to MXFP4 at load time so it can be fused into the routed experts; an already-MXFP4 shared expert is fused without the flag, so there it changes nothing. The fused slot runs through the routed experts’ W4A4 grouped GEMM, so the shared expert’s weights and its input activations are both quantized to MXFP4, where the standalone path it replaces ran in BF16; validate accuracy before using it.
  • KV cache offloading (MXFP4 on MI355X): Selecting Host DRAM (HiCache) adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits --enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first. HiCache extends the radix cache, so the generated command drops --disable-radix-cache — the two options are mutually exclusive and SGLang rejects them together at startup. --hicache-ratio 1.5 sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this Disabled for fixed-length throughput serving, where the extra host traffic buys nothing.
  • KV cache offloading (397B FP8 on MI355X): A separate recipe from the MXFP4 one above, for long-context agentic serving of the FP8 checkpoint. Selecting Host DRAM (HiCache) emits --tp 4 instead of the tp=2 that merely fits the weights, --enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first, --kv-cache-dtype fp8_e4m3 to halve device KV bytes, the SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1 env var so --mem-fraction-static 0.8 is not scaled down to 0.68, and the scheduler and tokenizer tuning --chunked-prefill-size 16384 --max-prefill-tokens 16384 --scheduler-recv-interval 60 --stream-interval 50 --tokenizer-worker-num 6. write_through mirrors every page to the host tier as it is written. --hicache-ratio 1.5 is the default sizing; the benchmarked high-concurrency points instead pin --hicache-size per rank (253 GB at concurrency 32-56, 374 GB at 64-80), which overrides the ratio — add it if you want a fixed host tier. This arm keeps --enable-aiter-allreduce-fusion and does not set ROCM_QUICK_REDUCE_QUANTIZATION; the MXFP4 recipe does the opposite. The full agentic recipe, including the GPU-resident concurrency 1-14 band that runs without HiCache, is in section 3.3.1. Read the FP8 KV cache warning below before using it for reasoning-heavy work.
  • Watchdog timeout: Increase --watchdog-timeout to 1200 or higher for this large model, as weight loading can take significant time.
  • Mamba Radix Cache: Qwen3.5’s hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via --mamba-radix-cache-strategy:
    • V1 (no_buffer): Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
    • V2 (extra_buffer): Enables overlap scheduling and branching point caching with --mamba-radix-cache-strategy extra_buffer --page-size 64. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. --page-size must satisfy FLA_CHUNK_SIZE % page_size == 0 or page_size % FLA_CHUNK_SIZE == 0 (FLA_CHUNK_SIZE is currently 64).
  • The --mem-fraction-static flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
  • Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
  • To speed up weight loading for this large model, add --model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}' to the launch command.
  • CUDA IPC Transport: Pass --mm-feature-transport cuda_ipc to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower --mem-fraction-static or --max-running-requests.
  • Multimodal Attention Backend: Use --mm-attention-backend fa3 on H100/H200 for better vision performance, or --mm-attention-backend fa4 on B200/B300.
  • B200 (FP8): Add --flashinfer-allreduce-fusion-backend auto for optimized throughput on Blackwell.
  • B200 / B300 (FP8 MoE): Add --moe-runner-backend flashinfer_trtllm to route the fused MoE through the trtllm-gen kernels on Blackwell. This is not picked up automatically — SGLang’s auto-selection of flashinfer_trtllm is limited to the DeepSeek architecture family, so an FP8 Qwen3.5 MoE run otherwise falls back to the Triton MoE runner. It applies to the MoE sizes (397B-A17B / 122B-A10B / 35B-A3B); the dense sizes have no MoE layers. The NVFP4 recipes already set it.
  • B200 / B300 (FP8): Add --linear-attn-prefill-backend flashinfer to run GDN (linear attention) prefill on the FlashInfer CuTe-DSL kernel. SGLang auto-selects it only inside a narrow validated domain (a chunked prefill size of at most 8192 and a bfloat16 mamba state pool, among other conditions), which these FP8 recipes fall outside of, so the override has to be explicit. Requires CUDA 13+ and flashinfer >= 0.6.14 — SGLang raises at startup on SM100 with an older CUDA. Applies to every Qwen3.5 size, since all of them are hybrid GDN models.
  • For processing large images or videos, you may need to lower --mem-fraction-static to leave room for image feature tensors.
  • Hardware requirements:
    • BF16: ~397B parameters require ~800GB of GPU memory for weights.
      • H100 (80GB) requires tp=16 (2 nodes) since each rank needs ~100GB at tp=8.
      • H200 (141GB) runs with tp=8.
      • B200 (183GB) runs with tp=8.
      • B300 (275GB) runs with tp=4.
      • MI300X (192GB) runs with tp=8.
      • MI325X (256GB) runs with tp=4.
      • MI355X (288GB) runs with tp=4.
    • FP8: The FP8 quantized model requires ~400GB for weights, cutting memory in half.
      • H100 (80GB) runs with tp=8.
      • H200 (141GB) runs with tp=4.
      • B200 (183GB) runs with tp=4.
      • B300 (275GB) runs with tp=2.
      • MI300X (192GB) runs with tp=4.
      • MI325X (256GB) runs with tp=2.
      • MI355X (288GB) runs with tp=2.
    • FP4: The FP4 quantized model requires ~250GB for weights, cutting memory by almost 4x. NVFP4 (nvidia/Qwen3.5-397B-A17B-NVFP4-V2) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint with FP8 attention (amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2) for MI355X.
      • B200 (183GB) runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4)
      • B300 (275GB) runs with tp=2. (NVFP4)
      • MI355X (288GB) runs with tp=2 (use tp=4 for low concurrency). (MXFP4)
HardwareMemoryBF16 TPFP8 TPFP4 TP
H10080GB168N/A
H200141GB84N/A
B200183GB844 / 2 + EP2 (MTP)
B300275GB422
MI300X192GB84N/A
MI325X256GB42N/A
MI355X288GB42 / 4 (agentic)2 / 4 (MXFP4)
FP8 KV Cache: --kv-cache-dtype fp8_e4m3 quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is emitted for the MXFP4-on-MI355X recipe and for the 397B FP8-on-MI355X agentic recipe (see section 3.3.1; the command generator in section 3.1 emits it when Host DRAM / HiCache is selected) and is part of the 397B FP8-on-MI300X agentic recipe in section 3.3.2; in these recipes it is part of the tuned configuration. For every other combination the generated commands leave it out, so add it manually only if memory constraints require the trade-off.
  • Xeon CPU service configuration: Please refer to the Notes part in the serving engine launching section in the SGLang CPU server document to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

3.3 Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments of Qwen3.5-397B-A17B, each benchmarked in SemiAnalysis InferenceX and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.
  • Router. Each configuration was submitted behind one front door: Dynamo (dynamo.frontend with dynamo.sglang workers) or SGLang (sglang.launch_server, behind a router when there are several workers or DP-attention ranks).
  • KV offload. Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes --enable-unified-cache-external-linker.
  • Container image. Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

3.3.1 Agentic Long-Context on MI355X (FP8, MTP)

Agentic replay on MI355X serves Qwen/Qwen3.5-397B-A17B-FP8 with MTP on four GPUs (TP4), benchmarked on the lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109 image. Concurrency below counts live sessions, not requests. Concurrency 1-14, GPU-resident: the sessions fit the device KV pool, so no HiCache tier is added. Concurrency 16-80, HiCache: a host-DRAM tier below the GPU KV cache keeps more prefix cached and reaches higher concurrency than the device pool alone. This is the Host DRAM (HiCache) selection in the command generator in section 3.1. From concurrency 32 up, the host tier is pinned with --hicache-size (GB per rank), which overrides --hicache-ratio, so make sure the node has that much free CPU DRAM per server on top of the model weights. The braced values in the command follow concurrency. Max running is --max-running-requests, Graph max batch is --cuda-graph-max-bs-decode, and HiCache GB is --hicache-size:
Command
For concurrency 16-80, add the HiCache flags to the command above, and --hicache-size {hicache-size} where the table gives one:
HiCache flags
A few notes on the command:
  • --speculative-algorithm EAGLE is the same algorithm the command generator in section 3.1 emits as NEXTN; SGLang resolves NEXTN to EAGLE.
  • SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1 makes the AITER backend use --mem-fraction-static 0.8 as-is. Without it, AITER multiplies an explicit value by 0.85 when the context length is above 8192, so 0.8 runs as 0.68. With it set, the KV pool grows from 7.83M to 11.32M tokens per GPU (+45%). The reserve it skips holds non-static attention workspace, so lowering --mem-fraction-static is the first thing to try if you run out of memory on a different workload or GPU.
  • AITER allreduce fusion stays on, and ROCM_QUICK_REDUCE_QUANTIZATION is not set. The two are mutually exclusive, and quick all-reduce belongs to the MXFP4 recipe. On images older than v0.5.23, ROCM_QUICK_REDUCE_QUANTIZATION defaults to INT8, so set ROCM_QUICK_REDUCE_QUANTIZATION=NONE explicitly there.
  • Weights load with multiple threads by default, so --model-loader-extra-config '{"enable_multithread_load": true}' is not needed.
  • --kv-cache-dtype fp8_e4m3 carries the same accuracy caveat as the FP8 KV cache warning in section 3.2.

3.3.2 Agentic Long-Context on MI300X (FP8, MTP)

Agentic replay on MI300X serves Qwen/Qwen3.5-397B-A17B-FP8 with MTP on eight GPUs (TP8), benchmarked on the lmsysorg/sglang-rocm:v0.5.23-rocm10-mi30x-20261109 image. Concurrency below counts live sessions, not requests. Every concurrency runs GPU-resident, with no HiCache tier. The braced values in the command follow concurrency. Max running is --max-running-requests, and Graph max batch is --cuda-graph-max-bs-decode:
Command
For the expert-parallel variant, benchmarked at the same concurrencies, add --expert-parallel-size 8 to the command above. A few notes on the command:
  • SGLANG_USE_AITER_UNIFIED_ATTN=1 runs the MTP target verify on AITER’s unified_attention, which needs --page-size 16. Without it, target verify runs on the Triton extend kernel, which has no split-KV and dominates decode time at agentic context lengths.
  • --max-running-requests is twice the concurrency because concurrency counts live session trees, and a tree can have more than one request in flight; the decode graph batch equals the concurrency.
  • Without --kv-cache-dtype fp8_e4m3, concurrency 24 and 32 are bound by device KV capacity on MI300X. The flag carries the same accuracy caveat as the FP8 KV cache warning in section 3.2.
  • SGLANG_USE_AITER=1 is already the default in the ROCm image, and the benchmark ran with it. Set it explicitly outside the image: AITER allreduce fusion and the AITER MoE path need it. AITER_FLYDSL_FORCE is not set because it already defaults to 1 in the image’s AITER build.
  • --speculative-algorithm EAGLE is the same algorithm the command generator in section 3.1 emits as NEXTN; SGLang resolves NEXTN to EAGLE.

4. Model Invocation

NVIDIA: Deploy Qwen3.5-397B-A17B with the following command (H200, all features enabled):
Command
AMD: Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X):
Command
Note: To enable MTP (speculative decoding), append --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.

4.1 Basic Usage

For basic API usage and request examples, please refer to:

4.2 Vision Input

Qwen3.5 supports image and video inputs as a unified vision-language model. Here is an example with an image:
Example
Output Example:
Output

4.3 Advanced Usage

4.3.1 Reasoning Parser

Qwen3.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via reasoning_content in the streaming response. To disable thinking and use Instruct mode, pass chat_template_kwargs at request time:
  • Thinking mode (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
  • Instruct mode ({"enable_thinking": false}): The model responds directly without a thinking process.
Example 1: Thinking Mode (Default) Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via reasoning_content:
Example
Output Example:
Output
Example 2: Instruct Mode (Thinking Off) To disable thinking and get a direct response, pass {"enable_thinking": false} via chat_template_kwargs:
Example
Output Example:
Output

4.3.2 Tool Calling

Qwen3.5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass extra_body={"chat_template_kwargs": {"enable_thinking": False}}. Python Example (with Thinking Process):
Example
Output Example:
Output

4.3.3 Semantic Decisions

Qwen3.5-35B-A3B can answer typed questions about an input with a probability for every option through /v1/decisions, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains /v1/decisions, install a nightly build. With a server started by sglang serve --model-path Qwen/Qwen3.5-35B-A3B --host 127.0.0.1 --port 30000:
Example
See Decision models for the request reference, how the probabilities are computed, and the limits.

5. Benchmark

5.1 Accuracy Benchmark

5.1.1 GSM8K Benchmark

  • Benchmark Command
Command
Historical results below use the removed harness and are not measurements of the sgl-eval command above.
  • Test Result
Output

5.1.2 Full GSM8K with sgl-eval

Command
The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

5.1.3 MMMU Benchmark

  • Benchmark Command
Command
  • Test Result
Output

5.2 Speed Benchmark

Test Environment:
  • Hardware: H200 (8x)
  • Model: Qwen3.5-397B-A17B
  • Tensor Parallelism: 8
  • SGLang Version: main branch
Server Launch Command:
Command

5.2.1 Latency Benchmark

Command
Output

5.2.2 Throughput Benchmark

Command
Output

5.3 Vision Speed Benchmark

We use SGLang’s built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.

5.3.1 Latency Benchmark

Command
Output

5.3.2 Throughput Benchmark

Command
Output