[Rui Ren (rui-ren)]

Scope

Add packed GGML IQ2_XXS and Q8_0 CUDA matrix multiplication, with explicit reconstructed-operand/output/bias rounding, and a separate NVFP4 A16 operator with FP32 global scaling after accumulation. This is a standalone primitive contribution, not a full Bolt readiness or performance claim.

Current immutable implementation: 9c9c43f, following 1c1b300 and df5d045. Base: 648f708. Three signed-off commits; no amend, rebase or history rewrite.

Correction to the initial draft

The initial draft decoded weights and retained them as FP32 until final output rounding. Subsequent independent qualification found that the pinned b12x BF16 A16 path rounds each full reconstructed weight to BF16 before multiplication, and adds bias after rounding the matmul output. This is a real source-semantics distinction, not reference noise.

The follow-up corrects IQ2 and adds a separately registered Q8_0 primitive. It does not modify MatMulNBits, recurrence, or existing NVFP4 kernels. Prior df5d binaries, measurements and mismatch evidence remain separately preserved; they are not relabeled as validation of this correction.

The third commit adds a distinct NVFP4 A16 contract, leaving the existing generic NVFP4 operator and its defaults unchanged. Prior df5d and v2 artifacts remain separately preserved.

Operator ABIs

Both operators are com.microsoft, version 1, no attributes:

Operator	A	B, raw UINT8	Optional bias	Y
MatMulGGMLIQ2XXS	FLOAT/FLOAT16/BFLOAT16 [...,K], rank>=1	[N,K/256,66]	[N], same type as A	[...,N], same type as A
MatMulGGMLQ8_0	FLOAT/FLOAT16/BFLOAT16 [...,K], rank>=1	[N,K/32,34]	[N], same type as A	[...,N], same type as A

N and K derive from B; there are no external scales, zero points or padded-tail convention. Empty dimensions, including K=0, are supported.

Arithmetic contract

Let round_T denote round-to-nearest-even to A's element type:

1. Decode embedded FP16 scales and packed values/codebook in FP32.
2. Round the complete reconstructed weight to T. Do not round the scale alone.
3. Multiply and accumulate in FP32, with unspecified summation order.
4. Round the matmul output to T.
5. If bias is present, add it to the already-rounded output, using FP32 followed by a second rounding to T.

Y = round_T(round_T(A @ round_T(dequantize(B)).T) + bias).

Without bias, step 5 is omitted. FLOAT rounding is the identity; FLOAT16 is the dtype-generic extension. BF16 boundaries match the pinned b12x A16 source ordering; native b12x execution was not qualified on H100.

The direct packed kernels support decode and prefill without a full-weight expansion, persistent expanded weights, full-matrix dequantization scratch or affine requantization. This is an operator-level statement, not a complete-model footprint claim.

Separate NVFP4 A16 ABI and arithmetic

com.microsoft::MatMulNVFP4A16, version 1, no attributes, exactly four inputs:

Input	Type and shape
A	FLOAT16/BFLOAT16 [...,K], rank>=1
B	UINT8 [N,K/2], low-nibble-first E2M1
weight_scale	UINT8 [N,K/16], raw E4M3 bytes, not numerically converted UINT8
weight_scale_2	One-element FLOAT tensor, global multiplier

K is divisible by 16; Y is [...,N] in A's native type. Empty dimensions/K=0 are supported. There is no fused bias, activation quantization or reciprocal-scale convention.

For all M, including M65+:

local_weight = round_T(E2M1(B) * E4M3(weight_scale))  # Excludes global scale.
accumulator = FP32_matmul(A, local_weight.T)
Y = round_T(accumulator * FP32(weight_scale_2))

Finite local products are exactly representable in FP16/BF16. Do not round the accumulator before applying global scaling, and do not fold the global scale into reconstructed native weights. Optional bias is a separate native Add, after output rounding. BF16 follows explicit b12x A16 source ordering; FP16 is the analogous extension, not a claim about b12x automatic activation-quantization modes.

Small M reuses existing packed CUDA execution without fused bias. General M uses a local-only native weight scratch matrix, FP32 cuBLAS output, and a separate FP32-global/native-output epilogue—not repeated small-M launches. It does not persistently expand weights, but does allocate per-call dense weights and FP32 accumulator scratch for general M. The zero-full-expansion statement above applies to IQ2/Q8, not NVFP4 general M. CUDA12.8+ is required. No hardware FP4-by-FP4 or whole-package24GB claim.

Format fidelity and licensing

IQ2_XXS is not affine INT2. Its 66-byte block holds a little-endian FP16 scale and eight 8-byte records. Each record contains four canonical grid selectors, four seven-bit sign indices with parity-derived eighth signs, and one four-bit local scale:

d * (0.5 + local_scale) * 0.25 * grid[index,j] * sign[j].

Byte-wise loads avoid alignment assumptions at the two-byte header and 66-byte stride.

Pinned canonical references:

• GGML decoder
• Magnitude grid
• Sign table

The canonical grid retains its full MIT license and ggml-author notice in iq2_xxs_grid.h. Its 2,048 little-endian bytes independently match both the retained ModelOpt codebook and the b12x test oracle:

05826b5d3e472a3a78f196be62ac78acf81df0f909626e12ab9fa2a5d490dd54

Q8_0 stores a little-endian FP16 scale followed by 32 signed two's-complement INT8 values. We preserve the scale bits and decode d*q in FP32 before rounding the whole weight. No codebook is required.

Independent source-ordering qualification

Pinned distribution: b12x 1.5.0 (SM120/SM121-specific).

• _lib/intrinsics.py::q8_0_pair_to_bf16x2: FP16 scale to FP32, signed INT8 to FP32, multiply, then cvt.rn.bf16x2.f32.
• gemm/blockscaled/_iq2_xs_gemv.py::_decode_eight: FP32 scale/codebook multiply followed by BF16 reconstruction.
• gemm/bf16_gemv/_kernel.py::_dot_bf16x8: BF16 operands consumed in FP32 products/accumulation.
• gemm/blockscaled/_linear.py::blockscaled_mm: BF16 matmul output followed by result.add_(bias).

Source SHA256:

• _lib/intrinsics.py: 7b9a28dd3a4f6339c8fa9c15cdc327b8de23802f4b529cc8aded9b65f97f8e3b
• _iq2_xs_gemv.py: ac08b0d6fe65ff5de41c5e67bc7e96b86fcca839d29ed00c56eb0bff0fb44322
• _linear.py: 51e0b639705237411b8ee5274e7ae94d3a8cdf8dbe57a9ac07a380b776a3dbfd

The b12x capability gate returns false on H100/SM90; its source validator rejects A16 with A16 requires SM120/SM121. Public native API import additionally fails because CuTe cutlass is absent. No backend was installed/built and no b12x native execution is claimed. Its independent CPU test decoders can be used without importing production kernels.

NVFP4 A16 source ordering (same pinned distribution):

• _lib/intrinsics.py:2975–3003: E2M1 and E4M3 local native operands, excluding global scale.
• _lib/dense_gemm.py:1719–1721: local NVFP4 decode; :870: FP32 accumulator.
• gemm/blockscaled/_preparation.py:78–96,278: native operands, FP32 accumulator and global FP32 alpha.
• _lib/dense_gemm.py:1767–1770,3172–3184: load alpha, FP32 accumulator scaling, then native output conversion.
• gemm/blockscaled/_linear.py:892–902: native output followed by separate native bias addition.

Additional source SHA256:

• _lib/dense_gemm.py: 66c5d91863d0ddff6a2da7db52550df622f3dce4a178f05d02226d4588873f08
• gemm/blockscaled/_preparation.py: 41b8cd87e023dc96263fabda6972447cb994294b674562d02897a77527348add

Validation

Physical GPU: NVIDIA H100, SM90.

C++ red/green: 29 passed, zero skips

Two exact rounding regressions were built and run against the preserved df5d provider first: reconstructed-weight cancellation and output-before-bias rounding both failed. Both pass with the correction.

Final targeted suite:

• 15 IQ2 tests, including all canonical grid elements/signs/local scales, negative/zero/subnormal scales, all three activation types, decode/prefill, arbitrary leading dimensions, empty dimensions/K=0 for all types, exact rounding/cancellation, static/runtime errors and explicit CPU-only failure.
• 10 Q8 tests, including all signed byte values, non-BF16-representable source scales, full-weight rather than scale-only rounding, exact cancellation/output-before-bias boundaries, all three types, decode/prefill, empty dimensions/K=0, shape/type failures and no CPU fallback.
• 4 unchanged regressions: existing NVFP4 BF16 and three BF16 INT8 MatMulNBits tests.

MatMulGGMLIQ2XXS.*:MatMulGGMLQ8_0.*:MatMulBlockQuantizedFp4WeightOpTest.WeightOnlyScalesBiasBf16:MatMulNBits.BFloat16_8bits:MatMulNBits.BFloat16_Int8_Gemm_Cuda:MatMulNBits.BFloat16_Int8_Chunked_Uint8ZeroPoint

Bounded actual-checkpoint rows: 36 checks passed

Model: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB, immutable HF revision:

c6cc2e8ffa353403213d6b9b18aaf8eb684d705a

Read only 16 rows each from routed IQ2 up-projection, shared IQ2 up-projection and Q8 lm_head. Independently load the unchanged b12x/testing/{iq2_xxs,q8_0}_reference.py::dequantize_blocks CPU functions; round the full reconstructed weights to native dtype; compute FP32 matmul, native output rounding, and optional nonzero bias afterward.

Three tensors × three dtypes × M=1/M=15 × bias absent/present = 36 passed.
All 24 FP16/BF16 cases are bitwise exact. Worst FP32 absolute error is 1.5273690223693848e-7. CPU EP fallback and TF32 are disabled; each profile attributes exactly the tested packed operator to CUDA.

This proves selected-row primitive semantics, not full expert/layer/decoder/generation parity or b12x native execution. No checkpoint data, evidence or binaries are committed.

Final v3 C++ and actual-row qualification

56 C++ tests passed, zero skips, on physical H100/GPU1:

• All15 IQ2 and10 Q8 tests above.
• 9 new NVFP4 A16 tests: FP16/BF16 M1/M15/M65, K16 fallback and K128 packed path, rank1/arbitrary leading dimensions, raw subnormal/zero/negative/max-finite scales, nonunit global, cancellation/overflow when global is incorrectly folded into weights, no native accumulator rounding before global scaling, empty M/N/K, static/runtime validation, CPU-only failure.
• 19 unchanged generic NVFP4 regressions.
• 3 unchanged BF16 INT8 MatMulNBits regressions.

MatMulNVFP4A16.*:MatMulGGMLIQ2XXS.*:MatMulGGMLQ8_0.*:MatMulBlockQuantizedFp4WeightOpTest.*:MatMulNBits.BFloat16_8bits:MatMulNBits.BFloat16_Int8_Gemm_Cuda:MatMulNBits.BFloat16_Int8_Chunked_Uint8ZeroPoint

60 bounded actual-row CUDA checks passed on the final v3 runtime:

• Reran all36 unchanged independent IQ2/Q8 checks above.
• 24 new NVFP4 checks, first16 real rows of layer0 Mamba in_proj (K4096) and out_proj (K8192), FP16/BF16, M1/M15/M65, bias absent or separate nonzero native Add. Preserved exact raw E4M3 scales and source scalar FP32 global values. The independently implemented pinned source A16 formula is the oracle; not native b12x execution. All24 outputs are bitwise exact.
• All48 FP16/BF16 checks across the three formats are bitwise exact. FP32 IQ2/Q8 worst absolute error remains 1.5273690223693848e-7.
• Strict CPU-fallback disable and TF32 off. Profiles attribute the packed operator, and the separate Add when present, to CUDA.

The existing generic NVFP4 operator remains unchanged. As an explicit contract-distinction diagnostic, its M65 output differs from the new A16 oracle for471/1040 and469/1040 actual in-projection FP16/BF16 outputs, and543/1040 and528/1040 out-projection outputs. Those differences are not masked or relabeled as generic-source parity. Existing generic regressions still pass.

Build-artifact provenance caveat

An isolated v2 overlay compiled the changed IQ2 kernel/test/schema, new Q8 wrapper/kernel/test, environment/opset registrar and CUDA registrar, then relinked local copies of affected archives with retained read-only baseline objects. The original df5d overlay and original build remain unchanged.

Not an all-current-main rebuild. Unchanged runtime/provider objects remain retained baseline objects; unrelated latest-main kernel changes are not qualified.

Frozen v2 core SHA256:
3dabf96ee347cb39e3cd40aed079434c894808c4b63388453dbe09d7ce0e8ec1

Frozen v2 CUDA provider SHA256:
cf28d98d0fd6b8de675516930c1cd54c8f50c1833a8a3e7f687f901333838ed4

Scoped clang-format and git diff --check passed.

The separately frozen v3 overlay adds only new NVFP4 A16 wrapper/kernel/tests and affected schema/opset/environment/CUDA registrations; it retains v2 IQ2/Q8 objects and unchanged generic NVFP4 baseline objects. It is likewise not an all-current-main rebuild. No original/v2 artifacts are replaced. Their recorded artifact hashes were rechecked unchanged.

Frozen v3 core SHA256:
62cc0188ef19594c724b3e7f4047b8908fd5852164d56edfd3e08a24734fb4d8

Frozen v3 CUDA provider SHA256:
cfa1cb429a8ddee246ece778aa743533009aa428dbfc6c786be908bbfe2dc8fb

All three commits are signed off with Copilot coauthorship.

Latest upstream main observed at publication: 1988019b6e23ac4def4628cddd603c1bfb504fe8. A read-only three-way merge analysis found no conflict markers. The source branch was not rebased, and this is not build validation of the newer upstream main.

Explicit non-claims / remaining integration gates

Gate	Status in this PR
Native packed IQ2/Q8 standalone CUDA primitives	Tested as above
Separate source-ordered NVFP4 A16 CUDA primitive	Tested as above; general-M dense scratch disclosed
Full Bolt decoder numerical parity	NOT_RUN
Full Bolt text generation	NOT_RUN
Full Bolt vision/multimodal pipeline	NOT_RUN
Full-package 24GB residency/footprint target	NOT_RUN
End-to-end latency/throughput/performance	NOT_RUN
All-current-main rebuild/full regression suite	NOT_RUN
Native b12x backend execution	Unsupported/unqualified on H100

Existing generic NVFP4 source-ordering differences were independently identified; the new dedicated operator deliberately does not alter the generic operator or claim its source parity. Routing, activation functions, recurrence, processor/media handling, whole-package assembly and generation remain separate integration work.

Draft only; no merge requested. No full-model readiness or performance claim.

[Copilot]

🟡 Changes recommended

NVFP4 A16 can select a scalar path that violates its FP32-product contract and overflows for inputs with finite expected outputs.

1 open issue

• Scalar GEMV violates A16 FP32 product semantics for unsupported K · New

What changed in this PR

Adds three standalone CUDA quantized matmul operators with explicit rounding contracts, without changing existing MatMulNBits or generic NVFP4 behavior.

Changes:

• Adds direct packed IQ2_XXS and Q8_0 execution.
• Adds NVFP4 A16 with FP32 global scaling after accumulation.
• Registers schemas and kernels, documents contracts, and adds numerical and validation tests.

File	Description
onnxruntime/​test/​contrib_ops/​matmul_nvfp4_a16_test.cc	Tests NVFP4 arithmetic and validation.
onnxruntime/​test/​contrib_ops/​matmul_ggml_q8_0_test.cc	Tests Q8 decoding and rounding.
onnxruntime/​test/​contrib_ops/​matmul_ggml_iq2_xxs_test.cc	Tests IQ2 decoding and rounding.
onnxruntime/​core/​graph/​contrib_ops/​ms_opset.h	Registers three operator schemas.
onnxruntime/​core/​graph/​contrib_ops/​contrib_defs.cc	Defines contracts and shape inference.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nvfp4_a16.h	Declares NVFP4 operator and helpers.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nvfp4_a16.cu	Implements local decoding and output scaling.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_nvfp4_a16.cc	Validates inputs and dispatches NVFP4 execution.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_q8_0.h	Declares Q8 operator and launcher.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_q8_0.cu	Implements packed Q8 matmul.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_q8_0.cc	Validates and dispatches Q8 inputs.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_iq2_xxs.h	Declares IQ2 operator and launcher.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_iq2_xxs.cu	Implements packed IQ2 matmul.
onnxruntime/​contrib_ops/​cuda/​quantization/​matmul_ggml_iq2_xxs.cc	Validates and dispatches IQ2 inputs.
onnxruntime/​contrib_ops/​cuda/​quantization/​iq2_xxs_grid.h	Adds licensed canonical IQ2 grid.
onnxruntime/​contrib_ops/​cuda/​cuda_contrib_kernels.cc	Registers CUDA kernels.
docs/​OperatorKernels.md	Lists supported kernel types.
docs/​ContribOperators.md	Documents operator formats and arithmetic.

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

LGTM — MatMulGGMLIQ2XXS, MatMulGGMLQ8_0, and MatMulNVFP4A16 schemas, shape inference, CUDA kernels (including the CanUseMatMulBlockQuantizedFp4WeightMma gate), and unit tests look clean.

[Rui Ren (rui-ren)]

LGTM — MatMulGGMLIQ2XXS, MatMulGGMLQ8_0, and MatMulNVFP4A16 schemas, shape inference, CUDA kernels (including the CanUseMatMulBlockQuantizedFp4WeightMma gate), and unit tests look clean.

Do you think we need to wire the nvfp4 and q_8 kernel into MatMulBlockQuantizedFp4Weight and MatMulBlockQuantizedFp8Weight ?

[Rui Ren (rui-ren)]

Copilot resolve the merge conflicts in this pull request

[kunal-vaishnavi]

LGTM — MatMulGGMLIQ2XXS, MatMulGGMLQ8_0, and MatMulNVFP4A16 schemas, shape inference, CUDA kernels (including the CanUseMatMulBlockQuantizedFp4WeightMma gate), and unit tests look clean.

Do you think we need to wire the nvfp4 and q_8 kernel into MatMulBlockQuantizedFp4Weight and MatMulBlockQuantizedFp8Weight ?

I was about to comment this as well. Otherwise, we will start a pattern of introducing a new MatMul contrib op for each quantization type + activation type.

[kunal-vaishnavi]

How does IQ2 differ from the existing 2-bit support in MatMulNBits?

[Tianlei Wu (tianleiwu)]

LGTM — MatMulGGMLIQ2XXS, MatMulGGMLQ8_0, and MatMulNVFP4A16 schemas, shape inference, CUDA kernels (including the CanUseMatMulBlockQuantizedFp4WeightMma gate), and unit tests look clean.

Do you think we need to wire the nvfp4 and q_8 kernel into MatMulBlockQuantizedFp4Weight and MatMulBlockQuantizedFp8Weight ?

We do not need these operators. Please take a look at my exported bolt models.