[Model] Support Qwen3.8-Flash-Next

[peakcrosser7]

Purpose

support https://huggingface.co/Qwen/Qwen3.8-Flash-Next

How to run

vllm serve Qwen/Qwen3.8-Flash-Next
          --served-model-name qwen3.8-flash-next
          -tp 4
          --enable-prefix-caching
          --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'

Enable PLE offload(note PLE offload support is in #53899)

VLLM_PLE_CPU_OFFLOAD=1

Validation

• Without offload:

  • Weight formats: BF16, FP8, and NVFP4
  • Platforms: GB300, GB200, H200, and MI355X (BF16 only)
  • Parallel configurations: TP2, TP4

• With N-gram embedding offload:

  • Weight formats: BF16 and FP8 (NVFP4 is not currently supported)
  • Platform: GB200
  • Parallel configurations: TP2, TP4, and DP4+EP4

The validation results were provided by Inferact. See the vLLM recipe for details.

Note: This PR is being updated with the latest changes from main and may be unstable during review. Please use PR #53899 for builds in the meantime.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[ZJY0516]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85734 for commit ea5b4577ece1.

[zzw09773]

While bringing this up on gfx942 we hit an out-of-bounds read that traces back to CircularBufferSpec
being classified as a normal token-to-slot cache group. Filed the kernel-side guard separately as
#53982; this is the model-side half.

CircularBufferSpec.max_num_blocks_per_req() returns 1, and QSAMetadataBuilder computes the
ring's slot mapping itself (_build_circular_metadata → circular_qsa_slot_mapping, which uses
only block_table[req, 0]). But gpu_model_runner.py:7387-7390 maps KVCacheSpecKind.MAMBA to
SlotMappingMode.NONE and everything else to TOKEN_TO_KV_SLOT, and CircularBufferSpec is not a
MambaSpec subclass — it falls through get_kv_cache_spec_kind() to UNKNOWN. So the generic
per-group kernel runs on this group every step, indexing a 128-token-wide row with raw token
positions, and its result is then discarded.

Suggestion: let the spec declare its own mode (e.g. a slot_mapping_mode property on KVCacheSpec,
default TOKEN_TO_KV_SLOT, overridden to NONE on MambaSpec and on self-managing side caches)
rather than enumerating exceptions in the runner. That also removes one wasted kernel launch per
group per step.

One caveat if you take this: NONE makes get_block_table_width use token_alignment=None, which
narrows the row to 1. That is correct for circular_qsa_slot_mapping (column 0 only), but worth
double-checking nothing else depends on the 128-token-aligned width.

[zzw09773]

Two things, both aimed at the "which configurations work" table @vadiklyutiy asked for.

1. The AMD ple_layer cannot load the FP8 checkpoint's n-gram table

With Qwen/Qwen3.8-Flash-Next-FP8, startup fails because ngram_embedding.weight_scale has no
corresponding parameter. In that checkpoint the PLE n-gram table is stored as F8_E4M3 across 128
shards with a per-tensor scale (0.000199 in our copy); the AMD embedding path only accepts a BF16
table.

The two files are not symmetric on this. In this PR, nvidia/ple_layer.py (1170 lines) defines an
FP8 embedding method — create_fp8_weight_parameter, create_fp8_scale_parameter, a
QuantizeMethodBase subclass, and a _dequantize_embeddings helper — while amd/ple_layer.py
(1059 lines) contains none of those symbols.

Two ways to close it — we are running the first and have the second prepared:

• Dequantize at load — read the FP8 shards, multiply by the scale, store BF16. Simple, costs
  the memory difference and some load time. This is what we are serving with.
• Keep it FP8 and dequantize at lookup — port the NVIDIA method verbatim, i.e. register the
  FP8 weight + scale parameters and dequantize the gathered rows in _dequantize_embeddings.
  We have this prepared as a 7-edit patch against a clean amd/ple_layer.py; F.embedding on an
  fp8_e4m3 tensor works on ROCm, so nothing else needs to change. Happy to open it as a PR if
  that is the direction you'd prefer.

2. gfx942 data point for the compatibility table

You listed GB300/GB200, H200 and MI355X. Here is gfx942 / MI325X, which is a different ROCm
target (CDNA3, FP8 is FNUZ, and the published ROCm image's aiter prebuilts are gfx950-oriented):

Hardware	8× MI325X (gfx942), ROCm 7.2.3
Parallelism	TP8 with --enable-expert-parallel
Context	--max-model-len 262144
Speculative	MTP, num_speculative_tokens=3, acceptance length 3.32
Throughput	152 tok/s single stream; 676 tok/s aggregate at 8-way concurrency
Correctness	needle-in-haystack to 250k 3/3; zh-TW output, reasoning_content, qwen3_coder tool parsing all correct

One constraint worth putting in the table because it is arithmetic, not a bug:
pure tensor parallelism is not valid for this checkpoint. moe_intermediate_size = 640 = 5×128
and the FP8 weight_block_size is [128, 128], so TP2/4/8 give 320/160/80 per rank, none of which
is a multiple of 128. Expert parallelism is required. (The shared expert is in
modules_to_not_convert, i.e. BF16, so it is not subject to this.)

Getting there needed three ROCm-side fixes beyond the PLE one — the spec-decode metadata allowlist,
an aiter JIT race, and a slot-mapping out-of-bounds read (#53982; the model-side half of that one
is in a separate comment on this PR). Filed separately so they don't clutter this thread.

Reproduction

Serve Qwen/Qwen3.8-Flash-Next-FP8 on an AMD GPU (so the amd/ module tree is selected).
Startup fails on ngram_embedding.weight_scale having no corresponding parameter. (We have not
run the nvidia/ tree; it is the one that defines the FP8 embedding method.)

What we verified / did not verify

• Verified: the symbol asymmetry between the two ple_layer.py files in this PR; the load
  failure and that both remedies clear it; all numbers in the table.
• Not verified: the native-FP8 port under long-running production load — we validated it
  structurally (anchors, dry-run line count, idempotency, AST) but are currently serving the
  dequantize-at-load variant.
• Not verified: any non-FP8 checkpoint on gfx942.

[LucasWilkinson]

we should try to avoid such model specific code here if possible; we spent alot of effort flushing out the DSv4 model specific code

[LucasWilkinson]

nit: i think we can just make this an MRV2 only model so we dont need to thrash MRV1 so much

[peakcrosser7]

@jhsmith409 Thanks for your feedback. We will add PLE-offload support for NVFP4 quantization as soon as possible. Once available, you can retry using the NVFP4 checkpoint provided by Inferact. We will also investigate your reported issue and work on a fix.

[peakcrosser7]

@vadiklyutiy Thank you for the review. We have updated the PR description with the configurations we tested and validated. We will also address your code suggestions shortly.

[LucasWilkinson]

Following up on #53896 (comment) made a proposal here: peakcrosser7#6 on how we can bring this closer inline with current main. I havent had a change to iterate on _get_packed_kv_cache_groups (vibecoded function for now; going to bed) as much as i would like (heading to bed) but directionally I think this right.

[peakcrosser7]

@LucasWilkinson Thanks for the review and the PR. I’ll refer to your changes and consider how best to refine the implementation.

[ZJY0516]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85763 for commit d9f64bd7094d.

[peakcrosser7]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85764 for commit d69520dd4559.

[zzw09773]

On the ROCm path (models/qwen3_8_flash_next/amd/qsa.py), Qwen3_8FlashNextQSAFlashAttentionBackend
subclasses FlashAttentionBackend and does not override supports_batch_invariance(), so it
inherits return True from vllm/v1/attention/backends/flash_attn.py. But the QSA implementation
does not use FlashAttention: Qwen3_8FlashNextQSAFlashAttentionImpl.forward_qsa calls the model's
own qsa_sparse_paged_attention Triton kernel.

That kernel is not batch-invariant, and the reason is concrete — the split-K reduction depth is a
function of the number of query rows in the batch:

base_programs = q.shape[0] * k_cache.shape[2]        # q.shape[0] = query rows in this batch
small_profile_limit = 8 if block_m <= 8 else 4
if   base_programs <= small_profile_limit: block_n, target_splits, partial_warps = 16, 64, 4
elif base_programs < 32:                   block_n, target_splits, partial_warps = 16, 32, 4
elif base_programs <= 256:                 block_n, target_splits, partial_warps = 64, 8, 2
elif base_programs <= 512:                 block_n, target_splits, partial_warps = 64, 4, 2
else:                                      block_n, target_splits, partial_warps = 64, 1, 2
...
num_splits = min(max_useful_splits, target_splits)

num_splits sets how many partial accumulators _qsa_merge_splitk_kernel combines. Crossing any
of those thresholds changes the reduction tree, and therefore changes the floating-point rounding
of the same request's attention output. That is precisely what VLLM_BATCH_INVARIANT is supposed
to rule out.

Why this is currently latent, and why it will not stay that way

On this model the flag cannot be reached today: vllm/v1/attention/selector.py raises before the
attention backend is consulted, because the GDN/mamba backend does not declare batch invariance:

if envs.VLLM_BATCH_INVARIANT and not mamba_attn_backend.supports_batch_invariance():
    raise RuntimeError(
        f"VLLM batch_invariant mode is not supported for {mamba_attn_backend.get_name()}.")

So the incorrect claim is masked by an unrelated guard. The moment mamba/GDN gains batch-invariance
support (#48613 tracks exactly that), this model would start up with
VLLM_BATCH_INVARIANT=1 and silently produce batch-dependent results — the failure mode the
flag exists to prevent. A backend-level capability claim that is only true by accident of a
neighbouring guard is worth fixing before it becomes load-bearing.

Suggestions

1. One line, immediate: override supports_batch_invariance() to return False on
   Qwen3_8FlashNextQSAFlashAttentionBackend. AttentionBackend already defaults to False; the
   True is inherited only because the class reuses FlashAttentionBackend for its metadata
   builder.
2. If QSA should support the flag: make the profile selection independent of q.shape[0] when
   vllm_is_batch_invariant() is set — pin (block_n, target_splits, partial_warps) to one
   profile so num_splits no longer depends on batch composition.

Option 1 is the correct default: declaring False costs nothing but an error message, while
declaring True wrongly costs silent numerical divergence.

[peakcrosser7]

@zzw09773 Thank you very much for your review and suggestions. ROCm support in the current PR is not yet complete, and we will address the remaining gaps as soon as possible.

[vowstar]

Build break on sm_80, one hunk.

In csrc/libtorch_stable/ops.h, fused_gdn_decode_post_conv_mtp is declared under VLLM_ENABLE_FUSED_KDA_DECODE but used under VLLM_ENABLE_FUSED_GDN_DECODE. On sm_90+ both macros are set so nothing shows. On sm_80 only the GDN macro is set and the build fails with "not declared in this scope". Giving the function its own VLLM_ENABLE_FUSED_GDN_DECODE guard fixes it.

Tested on 5x CMP 170HX (sm_80), pipeline parallel 5, with PR #53899 at f561eca. Three more small patches were needed for PP5, all in model code:

where	what
v1/worker/gpu_worker.py	enable the PLE offload client only on rank 0 and drop the PP=1 validation, only rank 0 owns PLE layers
models/qwen4_exp/nvidia/model.py	skip hyper_connection_mixer weights on ranks where the mixer is None
models/qwen4_exp/nvidia/model_state.py	let non-first PP ranks disable ngram state instead of raising

With those, plain PP5 serving works. MTP with PP does not: boot dies with a CUDA illegal memory access in the target model once speculative decoding makes decode query_len 3, on an intermediate pipeline rank, in both the V2 generic speculator and the V1 dedicated proposer paths. This is consistent with the recipe supporting only single-node TP or TEP, but worth knowing PP+MTP fails at runtime rather than at config time.

[peakcrosser7]

/ci run

[github-actions]

✅ Triggered Buildkite CI #85787 for commit 2a4cd640ff1a.