[BBaditi]

Mechanism

In bounded host-cache offload mode (#33622), QMoEOffload::RunColdExperts fuses per-expert chunked demand loads (FileExpertStore::LoadExpertChunk and FinishLoadExpert) with FC1 and FC2 tile execution in a single parallel dispatch. Previously, the FC1 and FC2 stages indexed group slots in raw 0 .. group_size - 1 order, so a demand-missing slot (slot_ready == 0) at a lower slot index caused worker threads to spin on its readiness barrier before reaching already-cached ready slots (slot_ready == kReadyDoneVal).

This change adds BuildReadyFirstSlotOrder and RunReadyFirstFusedColdExpertGroup in qmoe_shared_layout.h to stably partition group slots before parallel dispatch so initially ready cached slots execute before demand-loading slots in FC1 and FC2. Each slot retains its original activation input, lease binding, intermediate buffer, output buffer, and reduction order. When miss_load_count == 0 (all cache hits), reordering is bypassed. Pre-dispatch handling of in-flight prefetches (BeginLoadExpert returning -1) and lease cleanup behavior are unchanged.

Base & Tested Configuration

• Base branch: Applies cleanly on top of [CUDA] Optimize INT4 QMoE offload prefill/decode and S=8 doorbell execution #33622 (commit 5855bd4); ready-first slot ordering is independent of CPU SIMD ISA.
• Tested configuration on Zen 2: Both baseline and candidate benchmark builds merged [CUDA] Optimize INT4 QMoE offload prefill/decode and S=8 doorbell execution #33622 with [CUDA/CPU] Add AVX2 INT4 cold-expert kernel and zero-cold doorbell short-circuit for QMoE offload #33624 (commit 2ced7eaf86764c6cf25e38c2ba083c54251e9a45) so cold-expert INT4 GEMMs on the AVX2-only AMD EPYC 7V12 host executed Avx2Int4GemmRange rather than ScalarInt4GemmRange. AVX2 was part of both benchmark builds, not a functional requirement of ready-first scheduling.

Scoped Benchmark Summary

Autoregressive greedy single-token decode on Qwen3-30B-A3B INT4 (full_benchmark.onnx, 48 layers, 128 experts/layer, top_k=8, hidden=2048, intermediate=768, block_size=32), 1x NVIDIA A100-SXM4-80GB + AMD EPYC 7V12 (8 SpinPool CPU threads), tail_substitution_threshold=0.0, skip_zero_cold_doorbell=0. Evaluated across N = 8 balanced pairs (2 complete 4x4 cyclic Latin squares across 4 prompts x 32 generated tokens = 124 post-first tokens/run, counterbalanced across AB/BA orders = 32 paired requests/condition):

Condition (N = 8 Latin-Square pairs, 124 post-first tokens/run)	Baseline Decode Wall (ms, mean ± SD)	Candidate Decode Wall (ms, mean ± SD)	Paired Wall Diff (Cand - Base, mean ± 95% CI)	Baseline Sustained (tok/s, mean ± SD)	Candidate Sustained (tok/s, mean ± SD)	Paired Throughput Diff (mean ± 95% CI)
Primary: adaptive_promo_on, gpu_slots=32, host_cache_slots=16 (2.04 GB), prewarm=1	3592.00 ± 72.82	3533.61 ± 61.54	-58.38 ± 27.70 ms (-1.62% ± 0.74%, 95% CI [-2.36%, -0.88%], p = 0.0016, 8/8 faster)	34.534 ± 0.690	35.101 ± 0.613	+0.567 ± 0.256 tok/s (+1.65% ± 0.77%, 95% CI [+0.88%, +2.42%], median +1.43%)
Cache-Hit Control: static, gpu_slots=32, host_cache_slots=95 (12.11 GB), prewarm=1	3470.58 ± 60.32	3452.16 ± 39.59	-18.42 ± 30.80 ms (-0.52% ± 0.88%, 95% CI [-1.40%, +0.37%])	35.738 ± 0.614	35.924 ± 0.410	+0.186 ± 0.316 tok/s (+0.53% ± 0.89%, 95% CI [-0.36%, +1.42%])

Output Parity & Validation

• Output parity: Across all 64 paired requests (2,048 generated token steps), generated token sequences and final-step float32 logits SHA-256 hashes matched between baseline and candidate.
• Cache-hit control: No statistically detectable regression in the tested cache-hit control (host_cache_slots=95, paired decode wall-time difference -0.52% ± 0.88%, 95% CI [-1.40%, +0.37%]).
• Unit & regression tests: Added QMoESharedLayoutTest.ReadyFirstFusedColdExpertGroupCoversMixedHitsMissesDuplicatesAndFailures (13/13 non-AVX512 C++ tests in QMoESharedLayoutTest.* on the pure submission branch, 15/15 on the combined AVX2 branch, and all non-skipped Python tests in test_qmoe_offload_v4.py and test_qmoe_shared_layout.py pass).

Limitations

• Reorders compute slots within mixed ready + demand-loading groups only; synchronous LoadExpertChunk I/O (pread + TransposeInt4Scales) still executes on the same SpinPool CPU worker threads, yielding 1.65% higher throughput (95% CI [+0.88%, +2.42%]) in the tested Qwen INT4/A100/Zen 2 configuration.
• Benchmark timings above were collected on the minimal ready-first binary (175145b1...); subsequent review-only hygiene edits (MSVC intrin.h guard, group_size <= MaxGroupSlots bounds check, test placement for clean apply on [CUDA] Optimize INT4 QMoE offload prefill/decode and S=8 doorbell execution #33622, and reverting an unrelated 2-line GroupLeaseGuard edit for scope isolation) were validated with the C++ and Python regression suites without re-running the timed benchmark campaign.

[Copilot]

🟡 Changes recommended

The fused-dispatch tests pass even with identity slot mapping in both FC1 and FC2, leaving the ready-first scheduling optimization unprotected against regressions.

1 open issue

• Add fused-dispatch tests for mixed-group ready-first slot ordering · New

What changed in this PR

This PR builds on #33622’s fused QMoE cold-expert loading and execution by scheduling cached, ready experts before demand-loading experts, while preserving slot bindings and reduction order.

Changes:

• Adds stable ready-first ordering for FC1 and FC2, bypassing reordering for all-cache-hit groups.
• Integrates the shared dispatcher into QMoEOffload::RunColdExperts.
• Adds concurrent tests covering mixed hits/misses, all hits, duplicate experts, and load failures.

File	Description
onnxruntime/​test/​contrib_ops/​qmoe_shared_layout_test.cc	Adds fused-dispatch correctness and failure-handling tests.
onnxruntime/​contrib_ops/​cuda/​moe/​qmoe_offload.cc	Routes fused cold-expert execution through the ready-first helper.
onnxruntime/​contrib_ops/​cpu/​moe/​qmoe_shared_layout.h	Adds slot partitioning and shared fused-dispatch helpers.

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

Hi BBaditi — we just rebased and updated the base branch tlwu/20261026/qmoe-offload-s8-int4-opt (#33622) on top of the latest main + #33450–#33575:

1. FileExpertStore is now 100% byte-preserving: We removed scale transposition (transpose_rows/transpose_cols) from FileExpertStore so cold-expert bank loads preserve exact source bytes and CPU INT4 cold-expert GEMMs read [N, K/32] scales directly (iv.scales_are_block_major = false).
2. Rebase suggestion: Could you git rebase origin/tlwu/20261026/qmoe-offload-s8-int4-opt (1e0a3910d7) onto your branch bbaditi/qmoe-ready-first (0e20eea77a) and ensure RunReadyFirstFusedColdExpertGroup in qmoe_offload.cc sets iv.scales_are_block_major = false when constructing qmoe_shared::Int4ExpertView? That will bring this PR's diff back down to its single 3-file commit (+406 / -69).

[BBaditi]

Hi BBaditi — we just rebased and updated the base branch tlwu/20261026/qmoe-offload-s8-int4-opt (#33622) on top of the latest main + #33450–#33575:

1. FileExpertStore is now 100% byte-preserving: We removed scale transposition (transpose_rows/transpose_cols) from FileExpertStore so cold-expert bank loads preserve exact source bytes and CPU INT4 cold-expert GEMMs read [N, K/32] scales directly (iv.scales_are_block_major = false).
2. Rebase suggestion: Could you git rebase origin/tlwu/20261026/qmoe-offload-s8-int4-opt (1e0a3910d7) onto your branch bbaditi/qmoe-ready-first (0e20eea77a) and ensure RunReadyFirstFusedColdExpertGroup in qmoe_offload.cc sets iv.scales_are_block_major = false when constructing qmoe_shared::Int4ExpertView? That will bring this PR's diff back down to its single 3-file commit (+406 / -69).

Rebased onto the updated  tlwu/20261026/qmoe-offload-s8-int4-opt  branch ( iv.scales_are_block_major = false  preserved on  Int4ExpertView ) and added deterministic slot-order test coverage for  RunReadyFirstFusedColdExpertGroup

[Tianlei Wu (tianleiwu)]

Reviewed the ready-first scheduling map, FC1/FC2 slot identity, readiness publication, duplicate leases, failure cleanup, and caller integration. No actionable findings.

The current deterministic serial traces assert the actual FC1 and FC2 (slot, tile) sequence for mixed/all-hit/all-miss groups, addressing the existing thread about an identity-dispatch mutant passing. Threaded and load-failure coverage is retained. Complete current-head CI is green, so I did not repeat local runtime tests. This looks ready for manual approval; no approval submitted.

[Tianlei Wu (tianleiwu)]

BBaditi, there is still one feedback from Copilot not resolved. Please take a look whether it need fix.