Qwen · Benchmark run
Qwen3.6-27B Text NVFP4 MTP on RTX PRO 6000 Max-Q
Qwen3.6-27B-Text NVFP4 MTP served by vLLM reached 118.914 tok/s decode at a 4K shared repository prefix and 15.802 tok/s at a 261,504-token 256K-class shared repository prefix in the agentic-code hot-prefix benchmark.
Conditions
- Model
- Qwen3.6-27B-Text-NVFP4-MTP
- Family
- Qwen
- Parameters
- 27B
- Quantization
- NVFP4
- Rig
- windows-blackwell-max-q
- Topology
- Single GPU
- Nodes
- 1
- GPUs
- 1
- Tensor parallelism
- N/A
- Speculative decoding
- MTP=3
- Context
- 4,096 tokens
- Latency
- 138 ms
- Peak GPU temp
- 84 °C
- Engine
- vLLM
- Runtime version
- 0.23.0
- Operating system
- Windows (WSL2)
- Driver
- NVIDIA 596.72; vLLM Docker image vllm/vllm-openai:latest with CUDA 13.0 and Torch 2.11.0+cu130
Topology notes
Single RTX PRO 6000 Blackwell Max-Q GPU exposed to vLLM through Docker Desktop on Windows with WSL2 GPU passthrough.
Recipe
The 4K through 128K points use a 40G KV reservation. A 261,536-token shared-prefix attempt was rejected for publication because three of six requests failed after the task suffix plus 512 output tokens crossed the model service limit. vLLM also logged that Qwen3.5/Qwen3.6 Mamba align-mode prefix caching is experimental and emitted WSL pin_memory warnings.
Operator notes
The earlier 100 tok/s class NVFP4 result was not the best representation of this model because it did not use the downloaded ModelOpt MTP artifact and did not measure the agentic hot-prefix workload. The researched configuration uses --quantization modelopt, --language-model-only, vLLM generation config, automatic prefix caching, and MTP depth 3. The valid 4K through 128K ladder completed with zero failed requests at KV40G and max_num_batched_tokens 8192. Decode throughput moved from 118.914 tok/s at 4K to 27.979 tok/s at 128K, while MTP acceptance stayed roughly in the high 70 percent range. The top bucket needed real debugging. A 261,536-token prefix looked close to the model window, but task suffixes made some full requests too long, so that point had three failed requests and was rejected. A safer 261,504-token prefix left every task plus 512 output tokens under the 262,144-token limit. Reducing the KV reservation to 32G freed enough activation headroom, kept 470,789 GPU KV-cache tokens available, and completed all six 261,504-token requests with 15.802 tok/s decode, 15.263 tok/s full-duration output throughput, 80.667 percent speculative acceptance, and 3.42 mean accepted length. This is the important lesson for this model on a 96GB RTX PRO 6000: the highest reliable long-context result was not achieved by maximizing KV cache. At the 256K-class point, leaving activation room mattered more than reserving enough KV for extra concurrency.
Text generation sweep
Decode throughput at each measured context and concurrency, not just the headline number. Hover a point for its value, or toggle a series in the legend.