Your current environment
The output of python collect_env.py
Your output of `python collect_env.py` here
🐛 Describe the bug
Description
When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on long requests.
Environment
vLLM version: 0.19.1
Model: Qwen3.6-27B-FP8
TP size: 4, fp8 quantization, prefix caching + chunked prefill enabled
Speculative config: method=mtp, model=same as target, num_spec_tokens=5
Symptoms
The crash occurs after the request has accumulated ~26 k total tokens and has generated >1200 output tokens.
Just before the crash, spec metrics become abnormal: accepted tokens equal drafted tokens, acceptance rate jumps to 100%, and the scheduled draft tokens are all -1.
All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)).
Excerpt from logs
SpecDecoding metrics: Mean acceptance length: 6.00, ..., Avg Draft acceptance rate: 100.0%
scheduled_spec_decode_tokens={...: [-1, -1, -1, -1, -1]}
...
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
File "gpu_model_runner.py", line 1706, in _prepare_input_ids
prev_common_req_indices_tensor = torch.tensor(
To reproduce
Serve Qwen3.6-27B-FP8 with MTP speculative decoding using --num-speculative-tokens 5.
Send a conversation that grows to 25k+ tokens total.
The crash typically happens after 1000+ tokens have been generated.
Expected behavior
Generation should continue without invalid draft tokens or illegal memory access.
Additional notes
GPU memory usage was low (~5.4% KV cache), so it’s not an OOM issue.
The problem is reproducible; disabling speculative decoding avoids the crash.
Before submitting a new issue...
Your current environment
The output of
python collect_env.py🐛 Describe the bug
Description
When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on long requests.
Environment
vLLM version: 0.19.1
Model: Qwen3.6-27B-FP8
TP size: 4, fp8 quantization, prefix caching + chunked prefill enabled
Speculative config: method=mtp, model=same as target, num_spec_tokens=5
Symptoms
The crash occurs after the request has accumulated ~26 k total tokens and has generated >1200 output tokens.
Just before the crash, spec metrics become abnormal: accepted tokens equal drafted tokens, acceptance rate jumps to 100%, and the scheduled draft tokens are all -1.
All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)).
Excerpt from logs
SpecDecoding metrics: Mean acceptance length: 6.00, ..., Avg Draft acceptance rate: 100.0%
scheduled_spec_decode_tokens={...: [-1, -1, -1, -1, -1]}
...
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
File "gpu_model_runner.py", line 1706, in _prepare_input_ids
prev_common_req_indices_tensor = torch.tensor(
To reproduce
Serve Qwen3.6-27B-FP8 with MTP speculative decoding using --num-speculative-tokens 5.
Send a conversation that grows to 25k+ tokens total.
The crash typically happens after 1000+ tokens have been generated.
Expected behavior
Generation should continue without invalid draft tokens or illegal memory access.
Additional notes
GPU memory usage was low (~5.4% KV cache), so it’s not an OOM issue.
The problem is reproducible; disabling speculative decoding avoids the crash.
Before submitting a new issue...