[Bug][DSV4-Flash][DSpark] v0.27 weight loader regressed — KeyError routed_experts.w13_weight_scale (v0.26 gracefully skipped) #51916

Your current environment

• vLLM: 0.27.1 (pip, /data1/vllm027 venv)
• Python 3.11
• torch 2.13.0+cu130, triton 3.7.1, flashinfer-python 0.6.16.post3, transformers 5.15.0, CUDA 13.0
• Hardware: 8× NVIDIA H20-3e (141 GB), TP=8 + expert parallel
• (Full collect_env.py available on request.)

How to reproduce

Serve DeepSeek-V4-Flash-0731 with DSpark speculative decoding on v0.27.1 (the actual flags used):

vllm serve <DeepSeek-V4-Flash-0731-checkpoint> \
  --trust-remote-code \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --gpu-memory-utilization 0.85 --max-model-len 524288 \
  --max-num-batched-tokens 16384 --max-num-seqs 64 \
  --kv-cache-dtype fp8 --block-size 256 --dtype auto \
  --enable-prefix-caching --enable-chunked-prefill \
  --kv-offloading-size 512 --kv-offloading-backend native \
  --speculative-config.method dspark \
  --speculative-config.model <same Flash checkpoint> \
  --speculative-config.num_speculative_tokens 5 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 --disable-custom-all-reduce

Pre-req to reach this code path on Python ≤ 3.11: flashinfer 0.6.16's flashinfer/comm/fd_exchange.py:55 has a return annotation tuple[..., array.array[int]] that raises TypeError: type 'array.array' is not subscriptable at import time (array.array gains __class_getitem__ only in 3.12). That crash happens earlier and masks this bug. To reproduce this bug, either run on Python 3.12, or add from __future__ import annotations to flashinfer's fd_exchange.py.

Error

During draft-model weight load at startup (all TP/EP ranks fail identically):

  File ".../vllm/v1/worker/gpu/spec_decode/speculator.py", line 156, in load_model
    self.model = self.load_draft_model(target_model, target_attn_layer_names)
  File ".../vllm/v1/worker/gpu/spec_decode/dspark/speculator.py", line 81, in load_draft_model
    model = load_dspark_model(target_model, self.vllm_config)
  File ".../vllm/v1/worker/gpu/spec_decode/dspark/utils.py", line 45, in load_dspark_model
    draft_model = get_model(...)
    ...
  File ".../vllm/models/deepseek_v4/nvidia/dspark.py", line 449, in load_weights
    param = params_dict[name_mapped]
            ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'model.layers.0.ffn.experts.routed_experts.w13_weight_scale'

→ WorkerProc failed to start → engine init failed.

Root cause

v0.27's DSpark draft weight loader (dspark.py:449 load_weights) looks up the mapped name with a direct subscript params_dict[name_mapped]. The DSpark draft module does not register model.layers.0.ffn.experts.routed_experts.w13_weight_scale (a target-model routed-experts weight the draft does not carry), so the lookup raises.

This is a regression from v0.26: v0.26's loader skipped unknown draft weights with a "Skipping unknown DStorm weight ..." log line (graceful). v0.27 dropped that guard → hard KeyError.

Suggested fix

Restore the skip guard at dspark.py:449:

if name_mapped not in params_dict:
    continue
param = params_dict[name_mapped]

Safe because spec-decoding draft weights are validated by the target model; an unmapped draft weight should be skipped, not fatal. Restores v0.26 behavior.

Additional context

Two further issues block v0.27.1 on this exact stack (DSV4-Flash + DSpark + Hopper + Py3.11), tracked separately:

1. flashinfer 0.6.16 fd_exchange.py PEP-585 annotation crash on Py ≤ 3.11 (the pre-req above) — flashinfer side; one-line from __future__ import annotations fix.
2. After both fixes, the V1 engine hangs in warmup (shm_broadcast: No available shared memory broadcast block found in 60s repeating 7+ min, all 8 GPUs at 0% util). Root cause not yet isolated (may be a knock-on of the patched loader); not reporting yet pending a clean repro.

Happy to provide the full raw traceback / collect_env.py / test a fix on this 8×H20-3e stack.