Skip to content

[Bug]: MTP speculative decoding crash with illegal memory access on long sequences (Qwen3.6-27B-FP8, v0.19.1) #40756

Description

@SongXiaoMao

Your current environment

The output of python collect_env.py
Your output of `python collect_env.py` here

🐛 Describe the bug

Description
When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on long requests.

Environment

vLLM version: 0.19.1

Model: Qwen3.6-27B-FP8

TP size: 4, fp8 quantization, prefix caching + chunked prefill enabled

Speculative config: method=mtp, model=same as target, num_spec_tokens=5

Symptoms

The crash occurs after the request has accumulated ~26 k total tokens and has generated >1200 output tokens.

Just before the crash, spec metrics become abnormal: accepted tokens equal drafted tokens, acceptance rate jumps to 100%, and the scheduled draft tokens are all -1.

All worker processes then fail with torch.AcceleratorError: CUDA error: an illegal memory access was encountered in gpu_model_runner.py line 1706 (prev_common_req_indices_tensor = torch.tensor(...)).

Excerpt from logs

SpecDecoding metrics: Mean acceptance length: 6.00, ..., Avg Draft acceptance rate: 100.0%
scheduled_spec_decode_tokens={...: [-1, -1, -1, -1, -1]}
...
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
File "gpu_model_runner.py", line 1706, in _prepare_input_ids
prev_common_req_indices_tensor = torch.tensor(

To reproduce

Serve Qwen3.6-27B-FP8 with MTP speculative decoding using --num-speculative-tokens 5.

Send a conversation that grows to 25k+ tokens total.

The crash typically happens after 1000+ tokens have been generated.

Expected behavior
Generation should continue without invalid draft tokens or illegal memory access.

Additional notes

GPU memory usage was low (~5.4% KV cache), so it’s not an OOM issue.

The problem is reproducible; disabling speculative decoding avoids the crash.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions