You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR adds CUDA contrib operators for the packed NVFP4, Q8 and IQ2_XXS projections used by the existing Bolt model package. It moves these projections into ONNX Runtime without changing checkpoint values, re-quantizing weights, or replacing the source kernels with numerically different generic quantization operators:
Add three packed projection operators (matmul_packed_projection.cc, .h, .cu, .cuh, contrib_defs.cc, ms_opset.h, cuda_contrib_kernels.cc):
Register com.microsoft::MatMulPackedNVFP4, MatMulPackedQ8 and MatMulPackedIQ2XXS, with shape inference and runtime validation. Support BF16 activations/output and 1-8 flattened activation rows on CUDA SM80+, preserving leading dimensions. Reject unsupported shapes rather than silently selecting a different arithmetic path.
Preserve packed layouts and source precision boundaries (packed_projection_impl.cuh, matmul_packed_projection.cu):
Retain NVFP4's low-nibble-first E2M1 storage, E4M3 local scales and FP32 global multiplier; Q8's offset-binary bytes and separate FP16/BF16 scales; and IQ2_XXS's packed grid/sign records and raw FP16 super-scales. Preserve the specialized one-row paths, complete-operand BF16 rounding and source MMA/reduction topology, including cancellation-sensitive behavior at the N=8192 dispatch boundary.
Document format provenance and integration requirements (docs/contrib_ops/cuda/packed_projection.md, ContribOperators.md, OperatorKernels.md, ThirdPartyNotices.txt):
Record layouts, supported dimensions, frozen-source hashes and the IQ2 codebook's upstream MIT attribution. The operators do not provide CPU fallback, fused bias, or arbitrary-length prefill.
Address failures exposed by this PR's CI (qmoe_spin_pool_test.cc, tools/ci_build/requirements/transformers-test/requirements.txt, OperatorKernels.md):
Exclude SentencePiece 0.2.3, whose protobuf-7 generated schema is incompatible with the pinned protobuf 6.33.6 runtime. Remove a test-only /proc baseline-count race by tracking baseline thread IDs, retaining exact worker-count/join assertions and the existing timeout. Synchronize stale generated WebGPU Gather/Relu type rows with the provenance-verified CI artifact.
Exact current-head H100 regression: PASS.PackedProjectionKernelTest.Q8MmaReductionOrderCancellation was rebuilt from e54320acf0 and passed five executions with no skips.
CI-fix validation: PASS. All nine QMoE registry tests passed 20 repeats (180 executions). The disappearing-baseline regression fails with the original helper. The exact tokenizer dependency failure was reproduced; the compatible dependency combination loads and encodes with the actual T5 tokenizer without a tiktoken fallback. Isolated dependency checks, targeted lintrunner and whitespace checks pass.
Combined-port evidence: the earlier pinned overlay passed 24 direct CUDA-provider cases and 24 CUDA-graph capture/replay cases across all seven operators in the three-PR port, with CUDA placement confirmed and no Bolt custom-library registration. These are combined totals, not per-PR counts.
Qualification: focused compilation/linking reused cached unchanged dependencies; it is not a clean full build. Fresh-head CI is a separate acceptance gate. Fresh migrated-stack full-model throughput, MMLU/LiveCodeBench and independent HF parity are NOT_RUN; historical custom-stack results are not measurements of this PR. No lossless-speculation or release-readiness claim is made.
Stack:#33897 -> #33898 -> #33899. This PR targets main. Land in order, integrate newer predecessor fixes into the dependent branches/mirrors as needed, and rerun required main-branch CI after retargeting. The pre-existing ordinary-versus-k7 token discrepancy remains a separate qualification issue.
Port the released small-row NVFP4, Q8 and IQ2_XXS projection paths with schemas, CUDA registration, validation, tests and provenance. Preserve packed layouts, scale interpretation and source reduction/rounding; do not re-quantize weights.
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Test gaps leave key CUDA variants, FP16 accumulation behavior, and runtime row validation unverified, while the upstream license notice and generated operator documentation are also incomplete.
This first stack installment adds CUDA contrib operators for packed NVFP4, Q8, and IQ2_XXS projections. They preserve source layouts and arithmetic contracts for BF16 inputs/outputs with 1–8 flattened activation rows, without weight conversion.
Changes:
Add operator schemas, input validation, and CUDA registration.
Port packed decoding, accumulation, and rounding through shared kernel helpers.
Add focused tests, layout/provenance documentation, and per-source compilation flags.
Address the review findings while retaining the source layouts and valid-input arithmetic. Add focused boundary and correctness regressions and preserve each stack installment's existing scope.
Validation: changed-source host checks, CUDA SM80/SM90a compilation, and focused runtime regressions are recorded separately from clean-build and release qualification.
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The new rejection test covers the row limit but not the fixed-iteration K limits for NVFP4 and BF16-scale Q8. Add failures for row-one NVFP4 with K=1024 and BF16-scale Q8 with K=6208, each with and without graph shapes. These cases protect the guards that prevent out-of-range NVFP4 loads and omitted Q8 blocks.
Guard the NVFP4 device body for SM80+ with a trap fallback so default older CUDA targets compile without unsupported half2 intrinsics. Preserve the runtime SM80 requirement and all native arithmetic.
Add exact cancellation fixtures for R=3/K=1088/N=19 and 8192 with FP16 and BF16 scales in provider and CUDA internal tests. Exclude unavailable CUDA-minimal contrib kernels. CUDA 12.6 reproduces the old SM52 __hfma2 failure; corrected SM52/SM80/SM90a builds and the exact H100 launcher regression pass using this prefix's original shared header. Not a clean ORT/provider build.
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Constrain SentencePiece below 0.2.3: its protobuf-7 generated schema cannot load with the protobuf 6.33.6 runtime pinned by transformer CI. This prevents T5 tokenizer conversion from falling through to an irrelevant missing-tiktoken error on Linux and Windows.
Track baseline QMoE worker thread IDs rather than assuming a baseline count remains constant while joined threads disappear from /proc. Retain all creation and join assertions and the existing timeout, and add a deterministic disappearing-baseline regression.
Synchronize WebGPU Gather and Relu type rows with the provenance-verified updated-docs CI artifact. Production kernels and test thresholds are unchanged.
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With one identical corrected GenAI runtime in both arms, the current private
deployment versus the equivalent #33940 public route measured, for real
1028-1663-token inputs:
Ordinary: 92.02 -> 109.16 tok/s.
k=7: 318.91 -> 369.61 tok/s.
Both reversed-order passes completed 240 measured generations; full sequences
and within-mode counters matched, and packed-value/full-state checks passed.
This is not a fresh benchmark of the latest three predecessor heads together,
nor an isolated performance gain attributable to this individual PR.
#33940 remains an alternative/consolidation, not another PR to stack above #33897 -> #33898 -> #33899. This comment does not change this PR's source or its
original qualification. The common GenAI correctness source is published
separately as microsoft/onnxruntime-genai#2916;
the measured ORT gain is not attributed to R16 or a change in accepted drafts.
…xes (#2921)
## Summary
Complete ordinary target-only generation for native Nemotron Bolt while
preserving the existing fixed-eight-row target graph and greedy DFlash2
speculative path. Also fix configuration handling, shared-initializer
allocation metadata, and generation statistics.
The supported runtime remains batch-one, single-CUDA-device greedy
generation through the Model/Generator API, not paged Engine batching.
## Stack and dependencies
| Layer | PR | Scope | Base |
|---|---|---|---|
| 1/4 | #2920 | DFlash2 padding safety and API compatibility | `main` |
| 2/4 | **#2921** | Target-only generation, configuration, statistics,
and shared weights | #2920 |
| 3/4 | #2922 | Native drafter policy for partial-acceptance graph
replay | #2921 |
| 4/4 | #2923 | Verification arithmetic and opt-in one-token ordinary
decode | #2922 |
Merge in this order. After a parent lands, retarget its dependent PR to
`main` and rerun required CI. No merge is requested here.
**Current head:** `2e4886ef0046415dbbf583ee5822f087ff3c294c`.
This is the in-repository replacement for #2868. The original PR retains
earlier review/qualification history; the existing stack was not rebased
or rewritten.
## Changes in this layer
- **Target-only decoding:** accept `speculative.max_draft_tokens=0` or
`7`. A model configured with `0` skips drafter loading and drafter-only
state allocation. Selecting `0` on a model configured with a drafter
keeps its weights resident; these are different memory-measurement
configurations.
- **Capture controls:** allow ordinary decoding without target graph
capture. A capture request still requires a capture-enabled target
session; speculative decoding requires the compatible drafter and
capture-enabled sessions.
- **Configuration and public API:** reject unsupported package draft
counts before loading sessions, parse target-only configuration
independently of JSON member order, and allow the public speculative
setter's zero-draft mode for Bolt without changing unrelated model
behavior.
- **Optimizer settings:** add `GatherSliceToSplitFusion` to the
drafter's disabled optimizers without discarding caller exclusions.
- **Shared initializers:** use allocator memory metadata instead of the
CPU device-interface method that throws. Keep allocation identity
device-aware and test that shared buffers survive either session's
destruction, then expire after both sessions are gone.
- **State and statistics:** handle an absent drafter during
prefill/reset, report effective draft count zero in ordinary mode, and
make zero/partial/full acceptance buckets mutually exclusive. Cover EOS,
output budgets, correction/bonus tokens, and public-token commit/reset
behavior.
- **Runtime documentation:** clarify target-only versus speculative
execution, custom-op versus contrib-op packages, and the limits of
migration comparisons.
## Review fix included
`2e4886ef` declares the pinned `nlohmann_json` v3.12.0 dependency and
links `nlohmann_json::nlohmann_json` privately to `engine_unit_tests`.
The unconditional JSON include no longer relies on a system-installed
header.
The dependency fix is merged forward through #2922 and #2923 without
rewriting history. The corresponding review thread has been answered and
resolved.
## Validation and limits
- Native test configuration was attempted during the review fix, but
this Windows host has no `nmake` or C/C++ compiler. **No fresh local
native/GPU build is claimed for the review fix.**
- CI is **not yet green** at this description update; build failures and
pending jobs are visible in the [live
checks](https://github.com/microsoft/onnxruntime-genai/pull/2921/checks).
- This layer does not establish ordinary/speculative token equality,
independent HuggingFace parity, or a full accuracy/performance campaign.
#2923 adds a scoped arithmetic-boundary correction and opt-in ordinary
path; its qualification must not be attributed to this layer.
- Sampling, adaptive draft counts, dynamic batching, arbitrary rewind,
and multi-device execution remain outside scope.
<details>
<summary>Historical qualification and provenance</summary>
The original reconciled #2867/#2868 stack recorded eight
implementation/test units compiled, **16 CPU tests passed**, and **two
H100 tests passed**, using cached dependencies.
The CPU coverage included acceptance/budget logic, EOS, invalid
search/configuration rejection, target-only parsing/setter behavior,
optimizer exclusions, public-token reset, and shared-buffer lifetime.
The GPU tests covered shared-initializer lifetime in both
session-destruction orders and the predecessor's PagedAttention
padding/cache policy; they were not a new generation-throughput
campaign.
Original source anchor: `237754099f35f093c41e5d0cf3ce252d2dade854`.
These are historical combined-stack results, not a clean build or rerun
of current head `2e4886ef`.
The earlier custom-stack benchmark numbers in #2868 are historical, not
an isolated speedup from this PR. Its ordinary/k7 mismatch was not
resolved by this layer. Migration comparisons must distinguish `custom
k0 == contrib k0` and `custom k7 == contrib k7` from the stronger
ordinary/speculative equality requirement.
Plugin-free converted packages also depend on
microsoft/onnxruntime#33897, microsoft/onnxruntime#33898, and
microsoft/onnxruntime#33899. See #2868 for the original evidence and
runtime discussion.
</details>
---------
Signed-off-by: rui-ren <ruiren1225@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c8741c5a-3e9e-4860-b846-7d6fb0025040
Copilot-Session: f4d0e1a3-a389-4cc8-9c85-0fd0e3ffce5b
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds CUDA contrib operators for the packed NVFP4, Q8 and IQ2_XXS projections used by the existing Bolt model package. It moves these projections into ONNX Runtime without changing checkpoint values, re-quantizing weights, or replacing the source kernels with numerically different generic quantization operators:
Add three packed projection operators (
matmul_packed_projection.cc,.h,.cu,.cuh,contrib_defs.cc,ms_opset.h,cuda_contrib_kernels.cc):Register
com.microsoft::MatMulPackedNVFP4,MatMulPackedQ8andMatMulPackedIQ2XXS, with shape inference and runtime validation. Support BF16 activations/output and 1-8 flattened activation rows on CUDA SM80+, preserving leading dimensions. Reject unsupported shapes rather than silently selecting a different arithmetic path.Preserve packed layouts and source precision boundaries (
packed_projection_impl.cuh,matmul_packed_projection.cu):Retain NVFP4's low-nibble-first E2M1 storage, E4M3 local scales and FP32 global multiplier; Q8's offset-binary bytes and separate FP16/BF16 scales; and IQ2_XXS's packed grid/sign records and raw FP16 super-scales. Preserve the specialized one-row paths, complete-operand BF16 rounding and source MMA/reduction topology, including cancellation-sensitive behavior at the N=8192 dispatch boundary.
Share kernel helpers without changing existing operators (
packed_projection_impl.cuh,cmake/onnxruntime_providers_cuda.cmake):Expose the packed load/dequantization/MMA helpers for the follow-on routed MoE operator. Keep these contracts separate from existing GGML and NVFP4A16 primitives; [CUDA] Add source-ordered CUDA IQ2_XXS, Q8_0 and NVFP4 A16 matmul #33400 and Add sparse native IQ2_XXS CUDA execution to QMoE #33534 are not dependencies.
Document format provenance and integration requirements (
docs/contrib_ops/cuda/packed_projection.md,ContribOperators.md,OperatorKernels.md,ThirdPartyNotices.txt):Record layouts, supported dimensions, frozen-source hashes and the IQ2 codebook's upstream MIT attribution. The operators do not provide CPU fallback, fused bias, or arbitrary-length prefill.
Address failures exposed by this PR's CI (
qmoe_spin_pool_test.cc,tools/ci_build/requirements/transformers-test/requirements.txt,OperatorKernels.md):Exclude SentencePiece 0.2.3, whose protobuf-7 generated schema is incompatible with the pinned protobuf 6.33.6 runtime. Remove a test-only
/procbaseline-count race by tracking baseline thread IDs, retaining exact worker-count/join assertions and the existing timeout. Synchronize stale generated WebGPU Gather/Relu type rows with the provenance-verified CI artifact.Testing
MatMulPackedProjection.*covers nonunit scales, row counts 1/3/8, column tails, N=8192/8193 dispatch boundaries, multiple K chunks, NVFP4 K=4096/8192, Q8 scale variants, IQ2 raw-FP16 scale bits, and rejection of unsupported row counts.PackedProjectionKernelTest.Q8MmaReductionOrderCancellationwas rebuilt frome54320acf0and passed five executions with no skips.Stack: #33897 -> #33898 -> #33899. This PR targets
main. Land in order, integrate newer predecessor fixes into the dependent branches/mirrors as needed, and rerun required main-branch CI after retargeting. The pre-existing ordinary-versus-k7 token discrepancy remains a separate qualification issue.