Repository navigation
[CUDA] Consolidate source-exact BF16 decoding into existing contrib operators - #33940
Draft
Rui Ren (rui-ren) wants to merge 23 commits into
Draft
Rui Ren (rui-ren) wants to merge 23 commits into
Rui Ren (rui-ren) wants to merge 23 commits into
Conversation
Port the released small-row NVFP4, Q8 and IQ2_XXS projection paths with schemas, CUDA registration, validation, tests and provenance. Preserve packed layouts, scale interpretation and source reduction/rounding; do not re-quantize weights. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Port Mamba2Verify, BiasCorrectedMoERoute and ReluSquared with their existing shared implementation unit, schema contracts and tests. Preserve recurrent snapshots and arithmetic; commit/rollback policy remains with the caller. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the released routed expert path using the packed projection helper. Preserve complete-operand BF16 rounding, FC1/FC2 reductions, expert grouping and ordered weighted accumulation; include schema and focused tests. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Windows CUDA CI promotes MSVC C4244 to an error when the int64 element count expression implicitly becomes dim3's unsigned grid size. Express the existing conversion explicitly without changing launch geometry, kernel arithmetic or source precision. Reproduced the narrowing before the fix and compiled the corrected unit for SM80 and SM90a with host conversion warnings treated as errors. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…coverage Address the review findings while retaining the source layouts and valid-input arithmetic. Add focused boundary and correctness regressions and preserve each stack installment's existing scope. Validation: changed-source host checks, CUDA SM80/SM90a compilation, and focused runtime regressions are recorded separately from clean-build and release qualification. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The all-files lintrunner job also checks C++ formatting. Adding the standalone preprocessor branch changes automatic language-standard detection for the existing RunCuda nested-template spelling. Apply its one-line formatter output; no test semantics or kernel arithmetic change. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Promote convolution-weight offsets before multiplication and retain aligned vector loads with element-aligned scalar fallbacks. Use a single deterministic router comparator that places NaNs last. Preserve only trailing nonaliased state slots. Move the normalization-grid regression into the existing CUDA internal-test target. Add offset binding and independent K=768/E=511/512/topk=32 routing regressions, CUDA-minimal exclusions, and exact convolution-history tolerances. Validation: five host checks, exact production CUDA SM80/SM90a compilation, and four H100 internal-kernel regressions pass. Full linked ORT provider suite and full-model release qualification remain separate. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep aligned vector fast paths and use bit-preserving element loads/stores for offset X, packed FC1/FC2, hidden, expert output and Y bindings. Preserve decode, BF16 rounding, MMA and finalizer reduction order. Add an independent scalar-oracle R=8/topk=16/P=128 regression whose new experts first appear at pairs 0/32/64/96. Check complete metadata and all three numeric outputs exactly for aligned, six individual offsets and combined offsets. Exclude MoE provider tests in CUDA-minimal builds. Carry both predecessor fixes. Validation: 13 host/configuration checks, seven production CUDA SM52/SM80/SM90a builds, six H100 tests without skips, and separate original-header predecessor builds plus five H100 tests pass. Cached unchanged dependencies are reused; not a clean full build, linked provider-suite run or full-model qualification. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a warp-uniform finite-score fast path and exact precomputed ranks for exceptional scores. Preserve numeric-before-NaN ordering, stable expert-ID ties, removed-slot eligibility and the original FP32 weight arithmetic without repeating generic checks for every selection. Add independent-reference mixed-score coverage. H100 isolated tests and live routing comparisons pass; production-provider qualification is recorded separately before publication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Stage route indices once and distribute bounds and duplicate checks across route lanes instead of serially validating the complete row in every lane. Preserve whole-row exclusion with shared flags and a block-uniform exceptional-path masking barrier, without changing grouping order or MoE arithmetic. Add native coverage for invalid and duplicate routes, warp-boundary duplicates, offset bindings, metadata guards and NaN outputs. Both native tests pass against the production-compiled object; 120 full-model replay requests preserve all 60 paired complete output sequences and their archived same-mode references. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Dynamic alignment checks and byte/scalar fallback code in the packed-weight and activation load loop regress FC2 code generation even for aligned model buffers. Select an aligned template only after checking weight/hidden bases and the BF16 row stride at launch; preserve the existing unaligned fallback and FC1/dense defaults. Keep FP32 reconstruction, BF16 rounding, tensor-core math and reduction order unchanged. Extend native offset coverage to production intermediate sizes. H100 qualification passes 2,970 isolated bitwise comparisons and three production-compiled MoE/projection tests, with 60/60 complete paired generation sequences identical over the original 30 ordinary/k7 inputs. Actual in-model CUPTI FC2 latency falls from 38.004 to 31.756 us across 11,600 matched graph instances. This is local incremental-build qualification, not full-release acceptance. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The MoE offset-binding protection in 016af86 changed the shared activation loader and regressed NVFP4 code generation even on aligned model buffers. Select original predicated vector loads at launch only for NVFP4 MMA with aligned activation base and BF16 row stride. Retain the scalar unaligned fallback, row-one path, other formats and existing MoE/FC2 specializations. Weights, reconstruction precision and reduction topology are unchanged. Add native aligned/offset NVFP4 checks with nonunit scales, signed operands, production shapes, tails and output canaries. H100 qualification passes 384 isolated bitwise comparisons, four production-compiled native tests and 60/60 paired complete ordinary/k7 generation sequences on the original 30 questions. Actual target latency recovers from 28.536 to 25.420 us across 11,600 matched graph instances, versus archived original 25.429 us. This is local incremental-build qualification, not full-release acceptance. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reuse the cooperative grouping validation to initialize excluded hidden and expert-partial rows to NaN only on the exceptional path. Remove the unconditional invalid-row kernel and its redundant serial route checks without changing metadata, MMA arithmetic, or reduction order. Cover mixed/all-invalid routes, production intermediate dimensions, offset outputs, and buffer canaries. H100 qualification passes five native tests, 5,544 guard/metadata comparisons, 264 complete-operation bitwise comparisons, and 240 full-model requests. Two reversed-order passes show approximately 1.4% higher ordinary decode throughput; combined in-model guard/group latency falls from 7.397 to 4.137 us. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reuse the qualified private NVFP4 launcher through MatMulBlockQuantizedFp4Weight with gemv_kernel=fixed. Preserve default dispatch and reject unsupported fixed-mode inputs rather than changing arithmetic silently. Add cancellation, multirow/tail and invalid-selection regressions. This is a separate consolidation prototype; reference operator registrations are retained for controlled comparisons. Qualified on H100 with 22 CUDA tests, 640 unchanged default arrays, 640 exact fixed projections, 62 calibrated live-prefix projections, 1782 bitwise full-state comparisons and 240 exact-workload replay requests. Separate profiling confirms identical NVFP4 dispatch counts and effectively unchanged throughput. No GenAI changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Consolidate packed Q8 projections into existing MatMulNBits using an explicit fixed-dispatch profile and optional literal-FP16 scale input. Reuse the unchanged production launcher without repacking weights or allocating additional workspace. Keep ordinary MatMulNBits defaults unchanged and exclude the new profile from incompatible EP/fusion paths. Remove schema-injected dispatch metadata when constructing QLoRA nodes. Validation: 181 native passes, 30 existing skips, no failures; all five new cases executed. Production qualification passed 904 exact finite projection comparisons, 1782 full-state arrays (44489245824 bytes), and exact ordinary/k7 tokens and workload counters against the frozen control. The fresh GPU5 paired protocol passed all 240 requests / 120 pairs and separate lossless CUPTI with identical private projection dispatch. Across six bin/mode rows the measured throughput change is -0.490% to +0.157%; this is throughput preservation, not a claimed speedup. MIGraphX and OpenVINO rejection guards were linted but not compiled. No CUDA arithmetic or GenAI changes. Preserve original public-op PRs; removing obsolete registrations awaits combined-candidate qualification. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Reuse the qualified private MoE implementation through QMoE instead of requiring a separate public MoEPackedIQ2XXS node. Preserve FP16 super-scales, INT32 expert IDs, ordered FP32 mixing and invalid-row behavior; reject unsupported legacy routing and execution-provider combinations. Keep default integer and floating-point quantization behavior unchanged. Validate with native regressions, all 40 production expert banks, in-place offset/guard probes, 1,782 bitwise-exact full-state comparisons, exact ordinary/speculative generation, and a 240-request reversed-order same-H100 benchmark with separate CUPTI. Throughput is retained and private MoE kernel specializations/counts are unchanged. Regenerate the two extended operator documentation sections without unrelated documentation changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com> (cherry picked from commit 393dc1bafc617a20f4e4185f0f0032bfc19b466b) Signed-off-by: rui-ren <ruiren1225@gmail.com>
Extend the opt-in IQ2_XXS QMoE profile with optional router input, matrix, bias and scale. Require exactly one complete routing source while preserving legacy and preselected inputs. Reuse unchanged router and expert kernels without a separate public router node. Validate native and shape-inference regressions, all 40 production banks, 320 exact finite probes, 640 guarded input/output alias cases, 1782 exact full-state arrays and ordinary/k7 generation. Regenerate only QMoE documentation with the official schema generator. A separate same-GPU 240-request incremental ablation versus qualified preselected QMoE preserves exact tokens and workload counters. Observed ordinary throughput changes are -0.025% to +0.038%; k7 changes are -0.457% to -0.160%. Lossless CUPTI confirms unchanged compute specializations and counts. No speedup, independent HF parity or combined-candidate acceptance is claimed. No GenAI changes. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 2034f4c83911f23d726e1e8a0d3f52b1f43fca5b) Signed-off-by: rui-ren <ruiren1225@gmail.com>
Decode raw IQ2_XXS grid/sign/local-multiplier records through the existing CUDA MatMulNBits operator using an explicit fixed_iq2_xxs profile. Preserve literal FP16 super-scales, complete BF16 operand rounding, FP32 accumulation and final BF16 output rounding. Keep ordinary affine 2-bit quantization and default dispatch unchanged; reject unsupported profiles rather than silently falling back. Validate all four new native tests without skips, 185 passing native/internal regressions with 30 existing skips, 640 exact production comparisons across 80 banks, 1782 exact full-state arrays and identical ordinary/k7 generation and counters. Official schema-generated documentation changes only MatMulNBits. A same-GPU 240-request paired benchmark retains throughput (-0.011% to +0.078% observed), with lossless CUPTI confirming identical template4 compute specializations and counts. No meaningful speedup, independent HF parity or combined-candidate acceptance is claimed. No GenAI changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com> (cherry picked from commit d38e32a6459a3391a6bb592551d02c86ae329155) Signed-off-by: rui-ren <ruiren1225@gmail.com>
Allow independent FP16 or BF16 fixed Q8 scales without reinterpreting their dtype or changing CUDA arithmetic. Preserve the source row-one block-dot-before-scale path and its K limit; keep nonlinear IQ2 restricted to literal FP16 scales and default affine dispatch unchanged. Add five native regressions and regenerate only the MatMulNBits schema documentation. Native qualification passed 190 tests with 30 inherited skips; the actual draft LM-head bank matches bitwise for rows 1 through 8. Combined full-state, generation and performance qualification remain pending. No GenAI source changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com> (cherry picked from commit 3f184d002167abbc235e5b839f36a9688802bd50) Signed-off-by: rui-ren <ruiren1225@gmail.com>
Keep only Mamba2Verify as a new public schema. Route packed projections and IQ2 MoE through the qualified existing-op profiles, and remove obsolete schemas, CUDA registry entries and dead host wrappers without changing private CUDA arithmetic. Migrate operator fixtures while retaining independent hidden/expert/grouping and routing oracles, and regenerate schema/kernel documentation. Validation: 369 native cases passed with 43 inherited skips; all private oracle cases executed. Candidate-only collection on the cleaned production runtime matched 1782 complete frozen-control tensor hashes, all finite checks, ordinary/k7 token sequences and 882 counters. Scoped linters and whitespace checks passed. This is frozen-control equivalence, not independent HF qualification. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Rui Ren (rui-ren)
marked this pull request as ready for review
November 6, 2026 13:43
Rui Ren (rui-ren)
marked this pull request as draft
November 6, 2026 13:44
Keep the preselected-routing rejection in both native and plugin builds. Plugin Node adapters expose InputExists instead of the native InputDefs metadata API; use the corresponding API without changing CUDA arithmetic or default WebGPU routing behavior. Validation: compiled the exact production guard with real native and EP-adapter types under -Werror, reproduced the previous adapter compilation failure, and passed scoped lintrunner and whitespace checks. Full plugin builds are covered by CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical issues remain: bias fusion can cause valid fixed Q8/IQ2 graphs to fail initialization, and unaligned NVFP4/Q8 bindings can cause invalid CUDA accesses or incorrect loads.
3 open issues
What changed in this PR
This PR consolidates source-exact BF16 decoding into existing ONNX Runtime contrib operators, retaining Mamba2Verify as the only new public operator instead of the seven proposed in the reference stack.
Changes:
- Adds explicit fixed-arithmetic projection profiles and IQ2_XXS QMoE routing while preserving default dispatch.
- Adds snapshot-aware Mamba verification and guards incompatible optimization and provider paths.
- Adds operator and CUDA-kernel regressions, interface documentation, and codebook attribution.
| File | Description |
|---|---|
| ThirdPartyNotices.txt | Adds IQ2_XXS codebook attribution. |
| onnxruntime/test/optimizer/matmul_nbits_mlp_fusion_test.cc | Tests preservation of independent FP16 scales. |
| onnxruntime/test/contrib_ops/qmoe_shape_inference_test.cc | Tests IQ2 routing input contracts. |
| onnxruntime/test/contrib_ops/packed_moe_op_test.cc | Tests IQ2 QMoE arithmetic and validation. |
| onnxruntime/test/contrib_ops/matmul_block_scaled_fp4_test.cc | Tests fixed NVFP4 arithmetic and rejection paths. |
| onnxruntime/test/contrib_ops/fused_state_space_op_test.cc | Tests Mamba snapshots, aliasing, and registrations. |
| onnxruntime/test/contrib_ops/cuda_kernels/packed_projection_test.cc | Tests projection offsets and reduction order. |
| onnxruntime/test/contrib_ops/cuda_kernels/packed_moe_test.cc | Tests grouping, offset bindings, and invalid routes. |
| onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc | Tests normalization limits and router ordering. |
| onnxruntime/core/providers/openvino/ov_versions/data_ops.cc | Rejects MatMulNBits independent-scale inputs. |
| onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc | Excludes independent-scale MatMulNBits nodes. |
| onnxruntime/core/optimizer/matmul_nbits_qlora_fusion.cc | Guards QLoRA fusion against fixed-scale profiles. |
| onnxruntime/core/optimizer/matmul_nbits_qkv_fusion.cc | Excludes fixed-scale profiles from QKV fusion. |
| onnxruntime/core/optimizer/matmul_nbits_mlp_fusion.cc | Excludes fixed-scale profiles from MLP fusion. |
| onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h | Defines the Mamba2Verify schema. |
| onnxruntime/core/graph/contrib_ops/contrib_defs.cc | Extends projection and QMoE schemas. |
| onnxruntime/contrib_ops/webgpu/moe/qmoe.h | Rejects unsupported routing inputs. |
| onnxruntime/contrib_ops/cuda/sequence/FusedVerification.md | Documents verification contracts and qualification scope. |
| onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc | Implements Mamba validation and snapshot handling. |
| onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.h | Declares shared state-space and routing launchers. |
| onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.cu | Implements Mamba and routing CUDA kernels. |
| onnxruntime/contrib_ops/cuda/quantization/packed_moe_impl.h | Declares the packed IQ2 MoE launcher. |
| onnxruntime/contrib_ops/cuda/quantization/packed_moe_impl.cu | Implements grouped expert computation and accumulation. |
| onnxruntime/contrib_ops/cuda/quantization/matmul_packed_projection.cuh | Declares packed projection dispatch. |
| onnxruntime/contrib_ops/cuda/quantization/matmul_packed_projection.cu | Implements fixed NVFP4, Q8, and IQ2 projections. |
| onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.h | Validates fixed MatMulNBits profiles. |
| onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.cc | Integrates fixed dispatch and bypasses prepacking. |
| onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc | Integrates IQ2 experts and fused routing. |
| onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.h | Stores fixed-dispatch selection. |
| onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cc | Integrates fixed NVFP4 execution. |
| onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc | Registers Mamba2Verify for CUDA. |
| onnxruntime/contrib_ops/cpu/moe/moe_quantization_cpu.cc | Rejects unsupported IQ2 and routing inputs. |
| docs/ContribOperators.md | Documents the expanded public interfaces. |
| docs/contrib_ops/cuda/packed_projection.md | Documents projection layouts and arithmetic contracts. |
| cmake/onnxruntime_providers_cuda.cmake | Sets precision-related projection compilation options. |
Review effort: Balanced
The fixture always has 72 routing slots. Represent them with std::array instead of InlinedVector so GCC 14 no longer diagnoses accesses against inline storage during optimized warning-as-error builds. Preserve every routing case, expected metadata, invalid-row assertion and production-size offset check; do not suppress warnings or change production CUDA code. Validation: reproduced the original optimized GCC 14 failure, compiled the updated full test translation unit with GCC 14 and Werror, rebuilt the private test provider with production flags, and passed all three H100 MoE oracle cases against the exact qualified production engines. Scoped lint and whitespace checks passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Exclude fixed Q8 and IQ2 profiles from MatMulNBits bias fusion so Level2 optimization retains the separate Add. Handle element-aligned external NVFP4/Q8 bindings with scalar loads where needed, preserving the original vectorized specializations for aligned bindings. Document the contract. Add optimizer graph and real Level1/Level2 CUDA session regressions, plus independent activation, weight and scale offset matrices for row-one and MMA dispatch. All three pre-fix negative controls reproduced the reported failures. The corrected runtime passed 47 native cases and six private CUDA cases under memcheck with zero errors. All 1,782 full-state hashes, 18 ordinary/k=7 sequences and 882 speculative counter rows match the frozen quantized control; all 26 aligned projection kernel instruction streams are identical. Scoped lint and whitespace checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
3 of 4 tasks
Rui Ren (rui-ren)
added a commit
that referenced
this pull request
Nov 7, 2026
Retain the source-exact causal convolution, selective recurrence and gated grouped normalization implementation from #33940 above the packed MoE layer. Preserve device-selected starting snapshots, post-token states, aliasing, trailing storage and invalid-selector behavior without adding routing or projection operators. Include native schema/kernel registration and focused provider/internal tests. Production host, CUDA, schema, registry and test compilation passed. Live CUDA execution is NOT_RUN because this session currently has no device access; retain the PR as draft pending full-source runtime qualification. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Rui Ren (rui-ren)
added a commit
that referenced
this pull request
Nov 7, 2026
Retain the source-exact causal convolution, selective recurrence and gated grouped normalization implementation from #33940 above the packed MoE layer. Preserve device-selected starting snapshots, post-token states, aliasing, trailing storage and invalid-selector behavior without adding routing or projection operators. Include native schema/kernel registration and focused provider/internal tests. Production host, CUDA, schema, registry and test compilation passed. Live CUDA execution is NOT_RUN because this session currently has no device access; retain the PR as draft pending full-source runtime qualification. Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: rui-ren <ruiren1225@gmail.com>
Contributor
Author
Qualified exact IQ2 expert-up optimization — review branch, not a PR-head updatePublished the actual validated commit Exact FC1-only diff against the previously qualified runtime: 31 insertions / 12 deletions in two kernel files. Launch-selected aligned activation loads retain the supported fallback; signs are applied to the full FP32 product before BF16 conversion. Raw scale precision, MMA/K partitions, reduction order, routing and activation boundaries are unchanged. FC2 was left unchanged.
The isolated FC1 median gain is 8.49%, not an 8.49% end-to-end gain. The actual rebuilt provider passed 16 packed-native tests, 17 GenAI contracts (no skips), 891 complete ordinary-decoder tensor comparisons, and 240 paired real-prompt generations with full tokens and counter dictionaries unchanged. The persistent provider path was separately loaded and checked against all 891 tensors. Paired CUDA device profiles had no dropped/invalid records; the IQ2 expert family changed from 3.4370 to 3.3053 ms per ordinary target pass and 5.3659 to 5.1615 ms per k7 target pass. The next FC2 even-pointer experiment was rejected: distinct routes regressed 6.12%, despite a repeated-route gain. It is not in the production commit. Actual new CUDA provider SHA256: Integration limitation: this qualified runtime uses a different implementation/file layout from this PR's |
This was referenced Nov 9, 2026
Merged
3 tasks done
Tianlei Wu (tianleiwu)
pushed a commit
that referenced
this pull request
Nov 12, 2026
## Description Adds `com.microsoft::Mamba2Verify` version 1 (CUDA), a snapshot-aware Mamba2 operator for Nemotron Bolt. Fuses causal convolution, selective recurrence, and gated grouped RMS normalization. A device INT32 selector chooses the starting snapshot; each processed token writes its post-token convolution and SSM snapshots so the caller can select the accepted/final state on the next invocation. The retained contract uses BF16 activations/convolution, FP32 recurrent state, readout and normalization coefficients, head_dim=64, d_state=128, and 1-16 token rows. It preserves state aliasing, trailing snapshots, eight-row specialization, and invalid-selector non-finite output behavior. Longer prefills require chunking. This layer adds no projection, routing, or expert operator and requires no custom-op plugin. **Dependency:** #33963 in [native stack #33967](https://github.com/microsoft/onnxruntime/stack/33967); the eventual landing target is `main`. The slot-aware single-row replay readout contract is included in this introducing layer. ## Summary of Changes ### Schema and Kernel Registration | File | Change | |------|--------| | `onnxruntime/core/graph/contrib_ops/fused_state_space_defs.h` | Define Mamba2Verify inputs/outputs, type constraints, and shape inference. | | `onnxruntime/core/graph/contrib_ops/contrib_defs.cc, onnxruntime/contrib_ops/cuda/cuda_contrib_kernels.cc` | Register the contrib schema and CUDA kernel. | ### Implementation, Tests, and Documentation | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/sequence/fused_state_space.cc` | Validate geometry/snapshots and dispatch the CUDA implementation. | | `onnxruntime/contrib_ops/cuda/sequence/fused_state_space_impl.{cu,h}` | Implement convolution, recurrence, readout, normalization, and snapshot writes. | | `onnxruntime/test/contrib_ops/fused_state_space_op_test.cc, onnxruntime/test/contrib_ops/cuda_kernels/fused_state_space_test.cc` | Exercise provider-level state/selector behavior and internal CUDA kernels. | | `docs/contrib_ops/cuda/mamba2_verify.md, docs/ContribOperators.md, docs/OperatorKernels.md` | Describe the operator ABI, snapshot behavior, and CUDA support. | ## Testing ### Optional-output update - `3e40c0d4380572c4edd2aaf47b32883437bba418` - `xbc` (output 3) and `readout` (output 4) are independently optional; omitted intermediates use stream-aware ORT scratch workspace. Required outputs and all existing output positions are unchanged. - The scope remains one Bolt sequence per invocation. The public SSM layout remains `[slots, heads, d_state, head_dim]`. CUDA math, dtypes/cast boundaries, timestep clamping, state selection and the readout/replay guarantee are unchanged. - **PASS:** 19 repository gtests (5 schema, 10 CUDA provider, 4 internal CUDA) plus one fresh-kernel/schema guard: **20 checks, zero failures/skips**, on H100. Profiles record 66 `CUDAExecutionProvider/Mamba2Verify` node executions. - **PASS:** all four intermediate-output combinations, including an empty output-3 placeholder; state aliases, lower/upper invalid selectors and trailing/no-trailing snapshots. Eight capture cases (four combinations at one/eight rows) verify actual graph capture and bitwise replay. - **PASS:** 5 unwrapped CPU-only schema tests, compiled without `USE_CUDA` and run with `CUDA_VISIBLE_DEVICES` empty; zero failures/skips. - **PASS:** configured changed-range clang-format 20.1.8, whitespace checks and scoped official schema/kernel documentation rendering. The kernel-table row remains unchanged; generated schema documentation shows 3-5 outputs. Qualification freshly compiles the operator host/device code, schema and tests against ABI-matched retained ORT 1.32.0 infrastructure. It is **not a clean full-source build, full-CI approval, independent HF parity or full-model acceptance**. The initial mixed-infrastructure-header harness failure was diagnosed in `OpTester::BuildModel` and preserved; no production workaround or threshold relaxation was introduced. CUDA math/internal-test sources are byte-unchanged. ### Historical validation before the optional-output update **Review-fix snapshot:** `4f2bd30fe1f3017213ce9eb620b34ab507f5e392`. Fixed static rank/dimension validation with CPU-only schema tests, dt clamp/large-dt and NaN behavior, strict BF16 ULP/fraction checking and nonuniform 1024-wide normalization. The slot-aware replay correction and `MambaSingleRowReplayPreservesMultiRowReadout` regression are now owned by this introducing PR (moved from #34003), with both host references updated and the single-slot path preserved. The permanent contract is now `docs/contrib_ops/cuda/mamba2_verify.md`. **Owning-layer validation:** 8 provider + 3 CPU schema + 4 actual internal CUDA gtests; zero failures/skips. **Integrated tip regression:** 51 C++ provider/schema tests, 5 actual internal CUDA tests and 2 Python IOBinding tests passed, zero failures/skips. All owning-layer provider cases are retained. The final overlay recompiles the changed CUDA/host/schema/test sources over ABI-matched preserved ORT support; **not a clean full-source build, full-CI approval, independent HF parity or new end-to-end qualification**. Targeted configured clang-format/Ruff and whitespace checks passed. <details> <summary>Historical source provenance moved out of permanent documentation</summary> Origin: #33940, commit `491dc043952066267b2212ba979f0325da318573`, frozen external source `export-nemotron-pr-20261030/src/bolt_ops.cu`, SHA256 `1f587d9eacc356805c37d19e94e7aa4465dd50302576b2ca4b9579d7c5ae8ba5`. This identifies the original retained implementation, not a dependency in the merged tree or a promise that later dispatch fixes are absent. The slot-aware replay correction and current contract are documented in this PR and the permanent operator documentation. This is the Bolt precision contract, not independent HuggingFace model parity. </details> <details> <summary>Historical measurements and qualification evidence (not current-head acceptance)</summary> These previously reported results are retained for provenance. They have not been rerun on the rebased head above; earlier CI/build status and benchmark results do not establish current-head CI, accuracy, or merge readiness. ### Measured CUDA kernel latency - 2026-11-07 BF16 projection/convolution, FP32 SSM state; 128 heads, eight groups, head dimension 64, state size 128, channels 10240, projection stride 18560, four-tap convolution. One fused R-row production launch is compared with R serial single-row production calls with exact state handoff; every timed invocation starts from the same immutable initial state. This compares three fused kernels with 3*R serial kernels. Frozen **Bolt** precision contract: FP32 gate/normalization/gamma followed by BF16 output; this is not an acceptance test for the earlier Lightning cast-before-gamma contract. Conv snapshots, SSM snapshots and xBC are bitwise equal for all three cases in both runs. Y/readout pass normalized RMSE <=0.03 and cosine >=0.999; exact raw metrics are below. The one-row control is approximately unchanged; eight/sixteen-row fusion reduces latency. | Rows | Serial single-row calls us | Fused multi-row call us | Ratio | |---|---|---|---| | 1 | 8.91 | 8.91 | 1.00x | | 8 | 82.98 | 22.17 | 3.74x | | 16 | 167.01 | 41.41 | 4.03x | H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups, three graph warmups, and 15 paired timed batches alternating AB/BA. Four rotating allocation banks contain deterministic synthetic weights (seed 617); this is not a cold-cache guarantee or a checkpoint-weight benchmark. Raw p95 values describe batch-average latency, not individual-invocation tails. No profiler runs in the timing window. The assigned GPU had empty admission, one stable compute client during execution, and clean release afterward; clocks were not locked. Two independent 39-case runs passed, including the published replay runner. **Scope:** same-build production CUDA dispatch alternatives, not historical parent-commit binaries. The measured runtime is an isolated CUDA provider relink with preserved core/support, **not a clean full-source ORT build or end-to-end model qualification**. Each measured operator's CUDA source hash matches its owning PR head. A clean source build is a separate replication variant and must be measured again. Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB, speculative integration, or full-model readiness. [Complete primary/repeat results, numerical metrics, source hashes and benchmark code](#33969 (comment)). All 39 cases passed in both supervised runs; full-model work remains paused. No source push, stack change, merge, auto-merge or ready-for-review promotion is included. </details> ## Checklist - [x] Review feedback addressed with focused regressions - [x] Documentation and supported contracts updated - [x] Native stack order and immediate-parent dependencies preserved - [ ] Clean full-source/full-CI qualification (not claimed by focused overlays) --------- Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b
Tianlei Wu (tianleiwu)
pushed a commit
that referenced
this pull request
Nov 12, 2026
## Description Optimizes the existing CUDA MatMulPacked and IQ2_XXS MoEPacked paths without changing operator schemas, packed-weight layouts, or exported graphs. NVFP4 uses vectorized group decoding, paired BF16 operands, and half-chunk staging for eligible large-N Hopper projections. Q8 uses aligned packed-record windows with stride/alignment fallbacks; dense IQ2 reconstructs complete BF16 operands using the original K partitions. IQ2 experts use exact magnitude-codebook reconstruction and launch-selected aligned FC1 activation loads. FP32 scale reconstruction, sign-before-BF16 rounding, FC1/FC2 warp partitions, ReLU-square boundaries, and final reduction order are preserved. The shared LoadPacked8 fallback from the standalone-router extraction is reused. **Dependency:** #33994 in [native stack #33967](https://github.com/microsoft/onnxruntime/stack/33967). Mamba and standalone/integrated routing are unchanged. No requantization, GenAI inference change, graph conversion, or acceptance-threshold adjustment. ## Summary of Changes ### Packed Projection and Expert Kernels | File | Change | |------|--------| | `onnxruntime/contrib_ops/cuda/quantization/gguf_quant_cuda.cuh` | Add vectorized NVFP4/Q8 loads and dense IQ2 BF16 operand reconstruction. | | `onnxruntime/contrib_ops/cuda/quantization/matmul_packed.cu` | Select eligible optimized paths while retaining architecture/alignment fallbacks. | | `onnxruntime/contrib_ops/cuda/moe/moe_packed_iq2.cuh` | Add IQ2 expert codebook/operand helpers with exact FP32 scale reconstruction. | | `onnxruntime/contrib_ops/cuda/moe/moe_packed.cu` | Use the expert helpers and aligned FC1 activation-load specialization. | ### Regression Tests | File | Change | |------|--------| | `onnxruntime/test/contrib_ops/matmul_packed_precision_test.cc` | Cover production projection widths, NVFP4 operand extremes, Q8 record windows/stride fallback, and IQ2 complete operands. | | `onnxruntime/test/contrib_ops/moe_packed_routing_test.cc` | Add production-width and IQ2 codebook/scale reconstruction cases. | | `onnxruntime/test/contrib_ops/cuda_kernels/matmul_packed_kernel_test.cc` | Pin the compiled TC contiguous-128-K reduction for small/staged NVFP4 paths. | | `onnxruntime/test/python/transformers/test_gguf_packed_ops.py` | Exercise default-rounded byte-offset IOBinding fallbacks without a model precision attribute. | ## Testing **Session-policy compatibility update:** head `6294c1fd47b493fc5dbbbed6cfad3cc406ca476e`, rebased onto the default-rounded MatMulPacked contract from #33962. The removed node attribute is no longer emitted by test model factories or assigned through internal launch parameters. Default numerical references round weights for both activation dtypes and every dispatch. **Validation at optimized stack tip `a3f2076b65e0d1a5c7e9b82165ffab014e233876`:** 15 native CUDA + 15 compute75-only PTX projection tests, 5 Python format/policy/alignment tests, and 1 actual internal CUDA contiguous-128-K cancellation test; zero failures/skips. CLANGFORMAT, Ruff/format and whitespace checks pass. These are focused ABI-matched overlays with fresh changed objects, **not a clean full-source build, full CI, performance remeasurement, or new full-model qualification**. The previous broader review qualification and historical performance results are preserved below. <details> <summary>Historical measurements and qualification evidence (not current-head acceptance)</summary> ### Earlier review qualification (historical) **Review-fix snapshot:** `74847ad6021d5b5431fa5984efb7fa95f6a310cd`. Added offset-buffer IOBinding fallbacks (NVFP4/Q8 and IQ2 ReLU-square MoE), nonuniform single-product and bitwise 8-vs-9-row MoE oracles, all-256-position IQ2 one-hot coverage, NVFP4 high-nibble coverage and an actual internal contiguous-128-K cancellation regression. Removed duplicate IQ2 table/decoder logic and dead branches; lowered the operand-extremes fixture gate to SM80. Seven wrong-mapping mutations produce nonzero output differences in the new oracle. **Owning-layer validation:** 20 C++ + 2 Python IOBinding + 1 actual internal CUDA cancellation test; zero failures/skips. **Integrated tip regression:** 51 C++ provider/schema tests, 5 actual internal CUDA tests and 2 Python IOBinding tests passed, zero failures/skips. All owning-layer provider cases are retained. The final overlay recompiles the changed CUDA/host/schema/test sources over ABI-matched preserved ORT support; **not a clean full-source build, full-CI approval, independent HF parity or new end-to-end qualification**. Targeted configured clang-format/Ruff and whitespace checks passed. These previously reported results are retained for provenance. They have not been rerun on the rebased head above; earlier CI/build status and benchmark results do not establish current-head CI, accuracy, or merge readiness. Original comparison pins: parent #33994 at `a8ff5642f5c85519561964928006413a4fde7859`; candidate `1b5063c4af651e12bf955bc978c9312e2734a9f9`. **This is the actual stack-tip comparison**, not the earlier private-branch versus #33940 comparison. That distinction matters: the remote stack tip did not yet contain the optimized native kernels. | Input tokens | Stack ordinary tok/s | Candidate ordinary tok/s | Stack k=7 tok/s | Candidate k=7 tok/s | |---|---:|---:|---:|---:| | 178-254 | 54.36 | **92.42** | 223.84 | **355.42** | | 427-932 | 54.20 | **92.08** | 185.24 | **293.62** | | 1028-1663 | 54.25 | **92.04** | 201.48 | **318.91** | Approximately **70% ordinary** and **58-59% k=7** improvement in this controlled comparison. This does not claim that all remaining latency is resolved. The separate #33940 equivalent-package result of approximately 109 ordinary / 370 k=7 tok/s on the longest bin is **not** a measurement of this candidate; transferring that remaining gain is still outstanding. Protocol: original frozen 30 real prompts, ten per bin, unchanged warmups and per-prompt mode order, natural EOS/512-new-token cap, two reversed-order passes, same physical H100 per baseline/candidate shard. Eight logical shards ran in waves over GPUs 1,3,4,5,6,7. All admitted runs passed the foreign-client and resource guards. No profiler ran in the timing window; clocks were not locked. Rates pool decoded tokens over elapsed decode time, excluding the first token, not arithmetic means of per-request TPS. ## Correctness and build validation - Exact same native model package, packed bytes and GenAI libraries in both arms. Graph hashes match the preserved source-value audit; full package-file hashes were recorded again. No exported checkpoint/package was modified. - **22 native CUDA regressions passed, zero skips**, covering projections, complete operand rounding/cancellation, IQ2 production widths/codebook scales, routing and the standalone router. Changed C++ files passed CLANGFORMAT; git whitespace checks passed. - **1,782 complete live tensor/state hashes per arm**, bitwise equal and finite in ordinary and k=7 modes, also equal to the preserved quantized control. - Independent full LCB145 ordinary/k=7 probes passed all **512 generated tokens** and complete counters. - The accepted timing run passed **240 complete measured generation sequences** against frozen ordinary-control tokens. Within-mode complete speculative counter dictionaries matched between arms and repeats. - Isolated incremental rebuild on the archived clean #33994 build; every affected compilation unit/header dependent was rebuilt and the provider/test binary relinked with the production flags. Untouched objects were reused: **this is not a new clean full-source build**. - CUDA 12.8.93, driver 580.159.04, H100 80GB. Common corrected GenAI source: `c0fd01914265d5157d6da1376c38b4df521a76c5` ([microsoft/onnxruntime-genai#2916](microsoft/onnxruntime-genai#2916)). Both arms use that same source; its correctness fix is not credited with the kernel speedup. Accepted statistics SHA256: `0b9b7e2385b2dd9d7e5f7d459c65ef74338b5f707d1536454e7e27fb9a407c0f`. </details> ## Checklist - [x] Review feedback addressed with focused regressions - [x] Documentation and supported contracts updated - [x] Native stack order and immediate-parent dependencies preserved - [ ] Clean full-source/full-CI qualification (not claimed by focused overlays) ### Python IQ2 alignment regression correction Commit `f6e5af95c1` migrates `test_moe_packed_relu_iq2_aligned_input_fallback` from the removed seven-input compact-routing interface to six-input full-score routing: `[X, R, G, "", D, W]`, with FP32 selection scores and mixing values. Explicit `normalize_routing_weights=0`, buffer offsets and the exact bitwise oracle remain unchanged. No production code changes. The original test was reproduced failing graph resolution (7 inputs; schema maximum 6). The corrected actual Python unittest passed on H100 with zero skips, including all four combinations of rows 8/9 and activation offsets 0/2. Profiles confirm four `CUDAExecutionProvider/MoEPacked` executions. Ruff lint/format and whitespace checks passed. This run used the retained six-input full-score ORT 1.32.0 runtime built from `3bf5237cd65131701f2fb05435774672afaf1f96`; it is not a fresh runtime build of this PR, a full Python suite, or full CI approval. The corrected test file is byte-identical in the synchronized #34003 head. --------- Signed-off-by: rui-ren <ruiren1225@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Alternative to #33897, #33898 and #33899, preserving those PRs as references.
This branch consolidates their source-exact BF16 decode/verify implementation:
one new public operator, Mamba2Verify, rather than seven. It is an alternative
to the three-PR ORT stack, not another PR to stack on top of it.
The private CUDA arithmetic remains shared; reducing public schemas does not
mean removing the specialized quantized kernels.
Existing operators keep their default dispatch. The fixed profiles are explicit,
bounded opt-ins, with unsupported shapes/dtypes rejected rather than silently
falling back to different arithmetic. MatMulNBits uses optional independent-type
fixed_scales: genuine FP16/BF16 for Q8, literal FP16 only for IQ2. Existing
optimizer/EP paths are guarded from treating these profiles as affine codecs.
The six obsolete schemas and CUDA kernel registrations are removed, together
with dead host wrappers. Independent private hidden/expert/grouping and router
oracles remain. Generated schema/kernel documentation describes the actual
interfaces. A live registry test verifies that only Mamba2Verify remains among
the seven provisional names.
Validation
H100, CUDA 12.8.93, driver 580.159.04; isolated production core/CUDA/shared provider
hashes and loaded mappings recorded. No GenAI source changes or replacement
GenAI PRs are required for the qualified packages.
private CUDA oracles executed. A strict inventory check caught one stale
private-test object; it was rebuilt, and all three PackedMoE oracle cases then
ran against the exact production engines. The initial failure is preserved.
all 1,782 immutable seven-op-control tensor hashes, including complete logits,
hidden outputs and KV/conv/SSM states: 44,489,245,824 bytes, all finite.
all 882 speculative counter rows, with eight public generation steps per mode.
bindings were preserved. Scoped linters and whitespace checks passed.
independent HuggingFace qualification.
The final registration-cleaned production runtime passed the unchanged paired
benchmark: 240 fresh requests, 120 complete same-mode pairs, 20 requests per
variant per row. Full saved sequences and speculative counters matched.
Retains 98.36-98.79% of seven-op
control throughput. This is throughput retention, not the percentage of an
unmeasured generic-backend speedup. Separate lossless CUPTI captures had zero
dropped/invalid records. Only squared-ReLU dispatch changed; all other kernel
specializations and launch counts matched, including packed projections,
MoE, Mamba and attention. No second activation schema is added for the small
measured tradeoff.
Compatibility and provenance
Old graphs containing the six removed experimental operator names require the
reference PR runtime or conversion to the profiles above. This does not remove
an upstream-shipped operator. The original ORT and GenAI PRs are not changed by
this replacement.
Qualification uses pinned unchanged GenAI libraries with SHA256:
b32b2d599d569a3f8c67793465a2e2fd4dcef71fab6a80ca62c5585f95d9ef1c
86a0ddf92cb9ad63b263ae2f244c10f6a39b9dbd29094a712d4070872f4ebf1e
GenAI #2868 advanced independently to 237754099f35f093c41e5d0cf3ce252d2dade854
during this work. That newer head is not substituted for the frozen binaries
and is not covered by this qualification.
Local reproducibility/evidence root:
/root/.copilot/session-state/3b29a5de-77c7-4cf5-b77a-02adf4f73c89/files/
mapping, native XML/provider maps, candidate-only collector, full tensor hash
inventory and exact token/counter rows.
wrapper, all request sequences/counters, per-worker runtime mappings and
separate CUPTI captures/audit.
The qualified CUDA build source is ddda8d7.
The current PR head is 491dc04; its review corrections are described below.
Build receipts separately preserve the checkout base plus exact dirty-source
hashes; the final committed-tree receipt verifies every recorded input against
this commit. The original checkpoints, control packages and prior evidence are
not overwritten.
Final benchmark/audit SHA256: ccabae8e564854dc92c81bd5ca0beea2d3befbe8a061b0a35c58ba078d45dace
Committed-tree receipt SHA256: a5b25974b56d5f5551f1a181161281f75851478b7f584875f636ddf647630669
Follow-ups and current CI blocker
adapter API. Exact native/adapter guard compilation and a negative control
reproduced the previous incompatibility before the fix.
The original GCC 14 diagnostic was reproduced; the updated complete TU
compiles with optimization and -Werror. All three H100 MoE oracle cases,
including production-size invalid-row/offset coverage, reran with no skips
against the exact production engines. No warning suppression was added.
All five affected build checks passed on predecessor ce56c81: Windows WebGPU EP,
Windows WebGPU plugin, Linux static WebGPU plugin, Linux CUDA without cuDNN
linking, and Linux TensorRT. Those two follow-ups leave the qualified production binaries and CUDA arithmetic,
schemas, graphs, numerical and 240-request performance evidence unchanged. The review
corrections below use a fresh rebuilt runtime and separate validation evidence.
The PR remains draft; the review-correction head awaits CI. The predecessor's
React Native iOS E2E failure remains unwaived. Its single permitted retry passed MNIST inference and basic-types checks, but the third
test failed tapping the EPContext button because its label was not hittable.
That failure occurs before EPContext model checks run; it is not the first
attempt's simulator-setup timeout. The entire React Native subtree is
unchanged from upstream. Neither UI visibility requirements nor timeouts
were weakened, no unrelated UI fix was added, and no further retry is queued.
Failed retry: https://github.com/microsoft/onnxruntime/actions/runs/42234223915/job/128021304021
A separate upstream-identical spin-pool /proc thread-observation test failed
before shutdown in training CI. Its preceding-head pipeline and 800 isolated
local cases passed; the exact CI-only cause is not claimed fixed. Exactly one
training-job retry was requested; current result: success.
Training retry: https://github.com/microsoft/onnxruntime/actions/runs/42234224523/job/128019936613
Original and retry logs, result archives and local probes are preserved.
This is not a full-CI-green or merge-ready claim.
Review corrections in 491dc04
All three review findings are fixed: exclude fixed Q8/IQ2 profiles from bias fusion;
support element-aligned NVFP4/Q8 external bindings without imposing vector alignment;
retain original aligned-load specializations and document the binding contract.
Fresh H100 qualification: 47 native cases plus six private CUDA cases passed without
skips; the private cases ran under memcheck with zero errors. All three pre-fix
negative controls reproduced the reported failures. The corrected runtime matched
all 1,782 full-state hashes, 18 ordinary/k=7 sequences and 882 speculative counter
rows against the frozen quantized control. All 26 aligned projection kernels have
identical instruction text. No new throughput measurement or independent HF
qualification is claimed. GenAI source and original reference PRs are unchanged.
Evidence: operator-consolidation-public-cleanup/review-fixes-v5/ under the evidence
root above (BUILD_REPORT.json, VALIDATION_REPORT.v5.json, SASS_REPORT.json and raw
native/XML/memcheck/state outputs). Earlier failed admission, fixture and sanitizer
wrapper attempts are preserved separately. Existing benchmark numbers remain
measurements of the earlier qualified runtime. New-head CI is pending.
Common corrected-GenAI qualification (2026-11-08)
This is an additional controlled measurement of the public route in this PR
against the current private
MatMulPacked/MoEPackeddeployment, not anadditional 19% gain over this PR's older historical table, and not a new
measurement of the latest #33897 -> #33898 -> #33899 heads.
Both arms used the same corrected GenAI source
c0fd01914265d5157d6da1376c38b4df521a76c5(microsoft/onnxruntime-genai#2916),
equivalent graph I/O,
unchanged non-retargeted nodes and exactly equivalent effective weight values.
The GenAI correction was already present in both arms; it is not the source of
the speedup below.
Two reversed-order passes, ten original prompts per bin per pass, same physical
H100 for each native/public pair. Preserve original warmups, mode order, natural
EOS and the 512-new-token cap. Rates pool decoded tokens over decode time,
excluding the first token; they are not arithmetic averages of request TPS.
The final accepted run used GPUs 1,5,6,7. Earlier storage-guard and foreign-client
interruptions were preserved and excluded, not treated as accepted timings.
Correctness: all 240 complete measured generations matched the frozen
quantized ordinary-control sequences; complete speculative counter dictionaries
matched across compared runs. Complete IQ2/Q8/NVFP4 value fidelity, 1,782 full
tensor/state comparisons per variant (891 ordinary + 891 k=7), finite outputs
and the full 512-token LCB145 ordinary/k7 probe passed. The focused public native
regressions executed 47 cases with zero skips.
Separate actual-CUDA CUPTI captures (synthetic2048, not the timed real30 run)
attribute ordinary per-target-pass device work as follows:
Attention and Mamba timings were essentially unchanged. Routing is slightly
slower in this comparison; most net savings are in the matrix kernels.
All four captures had zero dropped/invalid records, matching complete generated
prefixes/counters and matching target-pass counts between arms. This bundle
comparison does not isolate the contribution of weight layout versus
shared-loop loading/pipelining.
Source/build scope and reproducibility
391f94e5a03950cd0bd9180fd5f18f75fd775c33.491dc043952066267b2212ba979f0325da318573.the unchanged corrected GenAI; did not claim a fresh full build, newly compiled
current-head MoE test fixture, or main-branch CI acceptance. The recorded
non-production source differences concern the WebGPU adapter and MoE fixture.
5859c053086adc11578bbfb618dfeb84d01447f570fea4bd99a9fe5acbd865b0.5ab9b0c7561a6fa2c4086b8bf9152dfec3d64942b7a8c1406e8bc7c08800ca4a.1f0b40efeefddc7f75b9c5ea18a85dc47a44e397df93cec8200163c4d7b97b8c.76d4aa528bc00aa0264c44415551a1df4f63147679ea853569f0f97cb0906cbd.521a0d9e8e92ab3c983152bf95fbc337bf3ab7411ed27a48b6f71e02b0249136.The faster public provider requires the matching public graph/packed-weight
package; it is not a drop-in provider replacement for private-op graphs.
The copied disk-persistent runtime also passed an actual-loaded-path and full
ordinary/k7 token probe. The existing public model package remains RAM-backed,
not reboot-persistent. No new independent HF accuracy, longer-context or
full-CI/merge-readiness claim is made. Original packages and earlier evidence
remain unchanged.