[Feature]: Support NVFP4 DeepSeek-V4-Flash-0731 with FP4 KV cache + DSpark speculative decoding on SM121 (DGX Spark)

[DamirLisak]

🚀 The feature, motivation and pitch

Summary

Requesting first-class support for serving NVFP4 DeepSeek-V4-Flash-0731 checkpoints (e.g. Rarri/DeepSeek-V4-Flash-0731-NVFP4) with FP4 KV cache (nvfp4 / nvfp4_ds_mla) and DSpark speculative decoding on SM121 (NVIDIA GB10 / DGX Spark, 2-node TP=2).

Today this combination does not work on vLLM main. We currently run it on a fork of jasl's branch (PR #41834) which serves the NVFP4 checkpoint but cannot do DSpark speculative decoding with it (draft loader fails with KeyError: 'layers.0.ffn.experts.routed_experts.w13_input_scale'). vLLM main v0.27.1 fails earlier at startup on SM121 with the DeepGEMM "Unknown SF transformation" error (see #51884).

Environment

• Hardware: 2× NVIDIA DGX Spark (GB10), compute capability 12.1 / SM121, Linux ARM64, two-node TP=2
• Model (NVFP4): Rarri/DeepSeek-V4-Flash-0731-NVFP4
  • ModelOpt mixed-precision conversion: producer: dsv4-nvfp4-experts-mtp-fallback
  • quant_method: fp8, quant_algo: MIXED_PRECISION, moe_quant_algo: NVFP4, group_size: 16, format: e4m3
• Model (FP8): deepseek-ai/DeepSeek-V4-Flash-0731
• vLLM versions tested:
  • v0.27.0 / v0.27.1 (main) — fails at startup on SM121 (DeepGEMM "Unknown SF transformation")
  • jasl fork branch codex/ds4-sm120-min-enable (PR [New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes #41834), image vllm-jasl-41834:latest — serves NVFP4 but no DSpark
• KV cache dtypes: fp8, fp8_ds_mla, nvfp4, nvfp4_ds_mla
• Speculative decoding: DSpark, 5 speculative tokens, --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}'
• Serving config: --gpu-memory-utilization 0.85, --block-size 256, MTP_NUM_TOKENS=5, MAX_MODEL_LEN=131072, MAX_NUM_SEQS=4

What we want

1. Serve the NVFP4 DeepSeek-V4-Flash-0731 checkpoint on SM121 with FP4 KV cache (nvfp4 / nvfp4_ds_mla).
2. Enable DSpark speculative decoding on top of that NVFP4 target (MTP draft), without the draft loader failing on missing NVFP4 scale tensors.
3. Have the DSpark draft built under its own model/quant config so an NVFP4 target does not corrupt the MXFP4 draft experts (see abandoned PR [Bugfix][Spec Decode] DSpark: build draft under its own model/quant config (NVFP4 target corrupts MXFP4 draft experts) #49133).

What fails today

On vLLM main (v0.27.1, official vllm/vllm-openai:v0.27.1-aarch64-cu129)

Startup fails on SM121 with the DeepGEMM scale-layout error:

RuntimeError: Assertion error (/workspace/.deps/deepgemm-src/csrc/apis/layout.hpp:60): Unknown SF transformation

This is the same class of failure as #51884 / #51758 (DeepGEMM SM120/121 regression). Full report: #51884.

On jasl's fork (PR #41834) — NVFP4 serves, but DSpark fails

The NVFP4 checkpoint loads and serves, but enabling DSpark fails during draft weight loading:

KeyError: 'layers.0.ffn.experts.routed_experts.w13_input_scale'

The draft loader expects a scale tensor that is absent from the NVFP4 draft params, so speculative decoding cannot be enabled for the NVFP4 target.

Related issues / PRs

• [Bug]: FP8 block-scaled weights fail on sm120 (RTX 5090) — DeepGEMM "Unknown SF transformation" during process_weights_after_loading #51884 — DeepGEMM "Unknown SF transformation" on SM120/121 with DeepSeek-V4-Flash FP8 + DSpark (blocker on main)
• [Bug]: upgrade vllm from 0.26.0 to 0.27.0 run deepseek v4 flash error #51758 — v0.26→v0.27 DeepSeek-V4-Flash load regression (DeepGEMM SM120)
• [Bug]: DeepSeek-V4-Flash-0731 + DSpark fails on RTX PRO 6000 (SM120) with FlashInfer sparse MLA decode kernel routing #50720 — DeepSeek-V4-Flash-0731 + DSpark fails on SM120 (FlashInfer sparse MLA topk=256 dispatch)
• [Bugfix][Spec Decode] DSpark: build draft under its own model/quant config (NVFP4 target corrupts MXFP4 draft experts) #49133 (PR, closed/abandoned) — "DSpark: build draft under its own model/quant config (NVFP4 target corrupts MXFP4 draft experts)". Closed by author with note: "we have nvfp4 drafter for dsv4-nvfp4 now" — but that drafter does not work for this checkpoint on SM121.
• [Bugfix][Spec Decode] Restore DeepSeek-V4 DSpark draft quantization #51835 (PR, open) — "Restore DeepSeek-V4 DSpark draft quantization" (restores deepseek_v4_fp8 for the draft)
• [New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes #41834 (PR) — "Add SM12x support for DeepSeek V4 Flash with essential fixes" (jasl's branch we currently run)
• [Feature]: Add DSpark speculative-decoding support for Rarri NVFP4 checkpoint jasl/vllm#43 — our original issue on jasl's fork for this exact scenario

Request

Please add/validate end-to-end support for NVFP4 DeepSeek-V4-Flash-0731 + FP4 KV cache + DSpark speculative decoding on SM121, including:

• Making the DSpark draft loader work with NVFP4 targets (missing w13_input_scale / NVFP4 scale tensors).
• Building the DSpark draft under its own model/quant config (revive [Bugfix][Spec Decode] DSpark: build draft under its own model/quant config (NVFP4 target corrupts MXFP4 draft experts) #49133) so NVFP4 targets don't corrupt MXFP4 draft experts.
• Confirming the nvfp4 / nvfp4_ds_mla KV cache path works with DSpark on SM121.

Happy to test any branch / provide logs.

[SyaOtiLan]

Tracking clarification: #51835 is being closed because its FP8 DSpark draft-quantization fix is now covered on main by #47808 / commit 70596c8.
That does not establish support for the broader NVFP4 scenario reported here. The NVFP4-specific w13_input_scale failure, separate draft/target quantization requirements, SM121 DeepGEMM support, and FP4 KV-cache validation remain distinct work.