1. Model Introduction
Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, representing a significant leap forward with unified vision-language foundation, efficient hybrid architecture, and scalable reinforcement learning. Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated), delivering high-throughput inference with minimal latency. It supports multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens. Architecture details:- Hybrid Attention: Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer — linear layers provide low-cost long-context processing while periodic full attention ensures high associative recall.
- MoE routing: Top-10 active out of 512 routed experts plus a dedicated shared expert for universal features, keeping 17B parameters active from 397B total.
- Native multimodal: DeepStack Vision Transformer with Conv3d temporal encoding for image and video understanding without separate visual encoders.
- Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models
- Efficient Hybrid Architecture: Gated Delta Networks + sparse MoE (397B total / 17B active) for high-throughput inference
- Hybrid Reasoning: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
- Tool Calling: Built-in tool calling support with
qwen3_coderparser - Multi-Token Prediction (MTP): Speculative decoding support for lower latency
- 201 Language Support: Expanded multilingual coverage across 201 languages and dialects
| Model | BF16 (Full precision) | FP8 (8-bit Quantized) | FP4 (4-bit Quantized) |
|---|---|---|---|
| Qwen3.5-397B-A17B | Qwen/Qwen3.5-397B-A17B | Qwen/Qwen3.5-397B-A17B-FP8 | NVIDIA NVFP4: nvidia/Qwen3.5-397B-A17B-NVFP4-V2 AMD MXFP4: amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 |
| Qwen3.5-122B-A10B | Qwen/Qwen3.5-122B-A10B | Qwen/Qwen3.5-122B-A10B-FP8 | - |
| Qwen3.5-35B-A3B | Qwen/Qwen3.5-35B-A3B | Qwen/Qwen3.5-35B-A3B-FP8 | - |
| Qwen3.5-27B | Qwen/Qwen3.5-27B | Qwen/Qwen3.5-27B-FP8 | - |
| Qwen3.5-9B | Qwen/Qwen3.5-9B | - | - |
| Qwen3.5-4B | Qwen/Qwen3.5-4B | - | - |
| Qwen3.5-2B | Qwen/Qwen3.5-2B | - | - |
| Qwen3.5-0.8B | Qwen/Qwen3.5-0.8B | - | - |
2. SGLang Installation
SGLang from the main branch is required for Qwen3.5. You can install from source or use a Docker image:Command
3. Model Deployment
This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.3.1 Basic Configuration
Interactive Command Generator: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- H100 / B200 / B300 FP8: Add
--enable-symm-memto enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. It only helps when tp > 1, so on B200/B300 it applies to the 397B-A17B FP8 recipe (tp=4); the 122B-A10B and 35B-A3B FP8 recipes are single-GPU there. - AMD GPUs (MI300X / MI325X / MI355X): Use
SGLANG_USE_AITER=1andSGLANG_USE_AITER_UNIFIED_ATTN=1with--attention-backend aiter, which requires--page-size 16and can also enable--enable-aiter-allreduce-fusion. Additionally setAITER_FLYDSL_FORCE=1to force the AITER FlyDSL MoE kernels andSGLANG_MAMBA_SSM_DTYPE=bfloat16to store the Mamba SSM state in bfloat16 (instead of the default float32). For the MXFP4 checkpoint on MI355X, setROCM_QUICK_REDUCE_QUANTIZATION=INT4to route multi-GPU collectives through INT4-quantized ROCm quick all-reduce, and drop--enable-aiter-allreduce-fusion(the two are mutually exclusive; quick all-reduce is preferred for this recipe). If the MXFP4 checkpoint keeps its shared expert unquantized,--enforce-shared-experts-fusionquantizes it to MXFP4 at load time so it can be fused into the routed experts; an already-MXFP4 shared expert is fused without the flag, so there it changes nothing. The fused slot runs through the routed experts’ W4A4 grouped GEMM, so the shared expert’s weights and its input activations are both quantized to MXFP4, where the standalone path it replaces ran in BF16; validate accuracy before using it. - KV cache offloading (MXFP4 on MI355X): Selecting Host DRAM (HiCache) adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits
--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first. HiCache extends the radix cache, so the generated command drops--disable-radix-cache— the two options are mutually exclusive and SGLang rejects them together at startup.--hicache-ratio 1.5sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this Disabled for fixed-length throughput serving, where the extra host traffic buys nothing. - KV cache offloading (397B FP8 on MI355X): A separate recipe from the MXFP4 one above, for long-context agentic serving of the FP8 checkpoint. Selecting Host DRAM (HiCache) emits
--tp 4instead of the tp=2 that merely fits the weights,--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first,--kv-cache-dtype fp8_e4m3to halve device KV bytes, theSGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1env var so--mem-fraction-static 0.8is not scaled down to 0.68, and the scheduler and tokenizer tuning--chunked-prefill-size 16384 --max-prefill-tokens 16384 --scheduler-recv-interval 60 --stream-interval 50 --tokenizer-worker-num 6.write_throughmirrors every page to the host tier as it is written.--hicache-ratio 1.5is the default sizing; the benchmarked high-concurrency points instead pin--hicache-sizeper rank (253 GB at concurrency 32-56, 374 GB at 64-80), which overrides the ratio — add it if you want a fixed host tier. This arm keeps--enable-aiter-allreduce-fusionand does not setROCM_QUICK_REDUCE_QUANTIZATION; the MXFP4 recipe does the opposite. The full agentic recipe, including the GPU-resident concurrency 1-14 band that runs without HiCache, is in section 3.3.1. Read the FP8 KV cache warning below before using it for reasoning-heavy work. - Watchdog timeout: Increase
--watchdog-timeoutto1200or higher for this large model, as weight loading can take significant time. - Mamba Radix Cache: Qwen3.5’s hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via
--mamba-radix-cache-strategy:- V1 (
no_buffer): Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. - V2 (
extra_buffer): Enables overlap scheduling and branching point caching with--mamba-radix-cache-strategy extra_buffer --page-size 64. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency.--page-sizemust satisfyFLA_CHUNK_SIZE % page_size == 0orpage_size % FLA_CHUNK_SIZE == 0(FLA_CHUNK_SIZEis currently 64).
- V1 (
- The
--mem-fraction-staticflag is recommended for optimal memory utilization, adjust it based on your hardware and workload. - Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
- To speed up weight loading for this large model, add
--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'to the launch command. - CUDA IPC Transport: Pass
--mm-feature-transport cuda_ipcto use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower--mem-fraction-staticor--max-running-requests. - Multimodal Attention Backend: Use
--mm-attention-backend fa3on H100/H200 for better vision performance, or--mm-attention-backend fa4on B200/B300. - B200 (FP8): Add
--flashinfer-allreduce-fusion-backend autofor optimized throughput on Blackwell. - B200 / B300 (FP8 MoE): Add
--moe-runner-backend flashinfer_trtllmto route the fused MoE through the trtllm-gen kernels on Blackwell. This is not picked up automatically — SGLang’s auto-selection offlashinfer_trtllmis limited to the DeepSeek architecture family, so an FP8 Qwen3.5 MoE run otherwise falls back to the Triton MoE runner. It applies to the MoE sizes (397B-A17B / 122B-A10B / 35B-A3B); the dense sizes have no MoE layers. The NVFP4 recipes already set it. - B200 / B300 (FP8): Add
--linear-attn-prefill-backend flashinferto run GDN (linear attention) prefill on the FlashInfer CuTe-DSL kernel. SGLang auto-selects it only inside a narrow validated domain (a chunked prefill size of at most 8192 and a bfloat16 mamba state pool, among other conditions), which these FP8 recipes fall outside of, so the override has to be explicit. Requires CUDA 13+ and flashinfer >= 0.6.14 — SGLang raises at startup on SM100 with an older CUDA. Applies to every Qwen3.5 size, since all of them are hybrid GDN models. - For processing large images or videos, you may need to lower
--mem-fraction-staticto leave room for image feature tensors. - Hardware requirements:
- BF16: ~397B parameters require ~800GB of GPU memory for weights.
- H100 (80GB) requires tp=16 (2 nodes) since each rank needs ~100GB at tp=8.
- H200 (141GB) runs with tp=8.
- B200 (183GB) runs with tp=8.
- B300 (275GB) runs with tp=4.
- MI300X (192GB) runs with tp=8.
- MI325X (256GB) runs with tp=4.
- MI355X (288GB) runs with tp=4.
- FP8: The FP8 quantized model requires ~400GB for weights, cutting memory in half.
- H100 (80GB) runs with tp=8.
- H200 (141GB) runs with tp=4.
- B200 (183GB) runs with tp=4.
- B300 (275GB) runs with tp=2.
- MI300X (192GB) runs with tp=4.
- MI325X (256GB) runs with tp=2.
- MI355X (288GB) runs with tp=2.
- FP4: The FP4 quantized model requires ~250GB for weights, cutting memory by almost 4x. NVFP4 (nvidia/Qwen3.5-397B-A17B-NVFP4-V2) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint with FP8 attention (amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2) for MI355X.
- B200 (183GB) runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4)
- B300 (275GB) runs with tp=2. (NVFP4)
- MI355X (288GB) runs with tp=2 (use tp=4 for low concurrency). (MXFP4)
- BF16: ~397B parameters require ~800GB of GPU memory for weights.
| Hardware | Memory | BF16 TP | FP8 TP | FP4 TP |
|---|---|---|---|---|
| H100 | 80GB | 16 | 8 | N/A |
| H200 | 141GB | 8 | 4 | N/A |
| B200 | 183GB | 8 | 4 | 4 / 2 + EP2 (MTP) |
| B300 | 275GB | 4 | 2 | 2 |
| MI300X | 192GB | 8 | 4 | N/A |
| MI325X | 256GB | 4 | 2 | N/A |
| MI355X | 288GB | 4 | 2 / 4 (agentic) | 2 / 4 (MXFP4) |
- Xeon CPU service configuration: Please refer to the
Notespart in the serving engine launching section in the SGLang CPU server document to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
3.3 Agentic Long-Context Deployment
Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments of Qwen3.5-397B-A17B, each benchmarked in SemiAnalysis InferenceX and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.- Router. Each configuration was submitted behind one front door: Dynamo (
dynamo.frontendwithdynamo.sglangworkers) or SGLang (sglang.launch_server, behind a router when there are several workers or DP-attention ranks). - KV offload. Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes
--enable-unified-cache-external-linker. - Container image. Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.
3.3.1 Agentic Long-Context on MI355X (FP8, MTP)
Agentic replay on MI355X servesQwen/Qwen3.5-397B-A17B-FP8 with MTP on four GPUs (TP4), benchmarked on the lmsysorg/sglang-rocm:v0.5.23-rocm720-mi35x-20261109 image. Concurrency below counts live sessions, not requests.
Concurrency 1-14, GPU-resident: the sessions fit the device KV pool, so no HiCache tier is added.
Concurrency 16-80, HiCache: a host-DRAM tier below the GPU KV cache keeps more prefix cached and reaches higher concurrency than the device pool alone. This is the Host DRAM (HiCache) selection in the command generator in section 3.1. From concurrency 32 up, the host tier is pinned with --hicache-size (GB per rank), which overrides --hicache-ratio, so make sure the node has that much free CPU DRAM per server on top of the model weights.
The braced values in the command follow concurrency. Max running is --max-running-requests, Graph max batch is --cuda-graph-max-bs-decode, and HiCache GB is --hicache-size:
Command
--hicache-size {hicache-size} where the table gives one:
HiCache flags
--speculative-algorithm EAGLEis the same algorithm the command generator in section 3.1 emits asNEXTN; SGLang resolvesNEXTNtoEAGLE.SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION=1makes the AITER backend use--mem-fraction-static 0.8as-is. Without it, AITER multiplies an explicit value by 0.85 when the context length is above 8192, so 0.8 runs as 0.68. With it set, the KV pool grows from 7.83M to 11.32M tokens per GPU (+45%). The reserve it skips holds non-static attention workspace, so lowering--mem-fraction-staticis the first thing to try if you run out of memory on a different workload or GPU.- AITER allreduce fusion stays on, and
ROCM_QUICK_REDUCE_QUANTIZATIONis not set. The two are mutually exclusive, and quick all-reduce belongs to the MXFP4 recipe. On images older thanv0.5.23,ROCM_QUICK_REDUCE_QUANTIZATIONdefaults to INT8, so setROCM_QUICK_REDUCE_QUANTIZATION=NONEexplicitly there. - Weights load with multiple threads by default, so
--model-loader-extra-config '{"enable_multithread_load": true}'is not needed. --kv-cache-dtype fp8_e4m3carries the same accuracy caveat as the FP8 KV cache warning in section 3.2.
3.3.2 Agentic Long-Context on MI300X (FP8, MTP)
Agentic replay on MI300X servesQwen/Qwen3.5-397B-A17B-FP8 with MTP on eight GPUs (TP8), benchmarked on the lmsysorg/sglang-rocm:v0.5.23-rocm10-mi30x-20261109 image. Concurrency below counts live sessions, not requests. Every concurrency runs GPU-resident, with no HiCache tier.
The braced values in the command follow concurrency. Max running is --max-running-requests, and Graph max batch is --cuda-graph-max-bs-decode:
Command
--expert-parallel-size 8 to the command above.
A few notes on the command:
SGLANG_USE_AITER_UNIFIED_ATTN=1runs the MTP target verify on AITER’sunified_attention, which needs--page-size 16. Without it, target verify runs on the Triton extend kernel, which has no split-KV and dominates decode time at agentic context lengths.--max-running-requestsis twice the concurrency because concurrency counts live session trees, and a tree can have more than one request in flight; the decode graph batch equals the concurrency.- Without
--kv-cache-dtype fp8_e4m3, concurrency 24 and 32 are bound by device KV capacity on MI300X. The flag carries the same accuracy caveat as the FP8 KV cache warning in section 3.2. SGLANG_USE_AITER=1is already the default in the ROCm image, and the benchmark ran with it. Set it explicitly outside the image: AITER allreduce fusion and the AITER MoE path need it.AITER_FLYDSL_FORCEis not set because it already defaults to1in the image’s AITER build.--speculative-algorithm EAGLEis the same algorithm the command generator in section 3.1 emits asNEXTN; SGLang resolvesNEXTNtoEAGLE.
4. Model Invocation
NVIDIA: Deploy Qwen3.5-397B-A17B with the following command (H200, all features enabled):Command
Command
Note: To enable MTP (speculative decoding), append --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4.
4.1 Basic Usage
For basic API usage and request examples, please refer to:4.2 Vision Input
Qwen3.5 supports image and video inputs as a unified vision-language model. Here is an example with an image:Example
Output
4.3 Advanced Usage
4.3.1 Reasoning Parser
Qwen3.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned viareasoning_content in the streaming response.
To disable thinking and use Instruct mode, pass chat_template_kwargs at request time:
- Thinking mode (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
- Instruct mode (
{"enable_thinking": false}): The model responds directly without a thinking process.
reasoning_content:
Example
Output
{"enable_thinking": false} via chat_template_kwargs:
Example
Output
4.3.2 Tool Calling
Qwen3.5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, passextra_body={"chat_template_kwargs": {"enable_thinking": False}}.
Python Example (with Thinking Process):
Example
Output
4.3.3 Semantic Decisions
Qwen3.5-35B-A3B can answer typed questions about an input with a probability for every option through/v1/decisions, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains /v1/decisions, install a nightly build. With a server started by sglang serve --model-path Qwen/Qwen3.5-35B-A3B --host 127.0.0.1 --port 30000:
Example
5. Benchmark
5.1 Accuracy Benchmark
5.1.1 GSM8K Benchmark
- Benchmark Command
Command
- Test Result
Output
5.1.2 Full GSM8K with sgl-eval
Command
5.1.3 MMMU Benchmark
- Benchmark Command
Command
- Test Result
Output
5.2 Speed Benchmark
Test Environment:- Hardware: H200 (8x)
- Model: Qwen3.5-397B-A17B
- Tensor Parallelism: 8
- SGLang Version: main branch
Command
5.2.1 Latency Benchmark
Command
Output
5.2.2 Throughput Benchmark
Command
Output
5.3 Vision Speed Benchmark
We use SGLang’s built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.5.3.1 Latency Benchmark
Command
Output
5.3.2 Throughput Benchmark
Command
Output
