Repository navigation
Conversation
Signed-off-by: David Fan <jiafa@microsoft.com>
|
|
This was referenced Oct 21, 2026
shantanusharma
pushed a commit
to shantanusharma/onnxruntime
that referenced
this pull request
Oct 22, 2026
) ## Summary Implements the P0 reference/tooling and P1 QMoE v1 contract/host-validation slices of the [INT3 QMoE delivery plan](https://github.com/microsoft/onnxruntime/blob/2b1185e24a5a79c56fd01226ad81b016450d0ef3/docs/design/int3-qmoe-end-to-end-delivery-plan.md). Related roadmap: microsoft#32657. No P2 CUDA kernels are included. ### P0: Portable Format and Numerical References - Generic row-local LSB-first UINT8 pack/unpack for widths 1..8, checked size arithmetic, and separate canonical-padding validation. INT5/INT6 utility coverage does not enable those widths in QMoE. - INT3 offset-4 decoding, deterministic fixture RTN with stored-scale rounding and subnormal-underflow reporting, and handwritten byte/numerical vectors covering row/expert boundaries, packed block zero points, and a tiny predetermined-route SwiGLU MoE. - Independent scalar oracles and standard opset-18 ONNX unpack/decode graphs; no draft Pack/Unpack operator is required. - Self-contained reference tests use local golden vectors and installed standard ONNX operators; no external ONNX source checkout or pinned ONNX commit is required. - Reference pytest coverage runs in the existing Python quantization CI stage; CMake copies the helper modules and golden JSON fixture into the build output. No feature-specific workflow is added. ### P1: Compatible QMoE v1 Extension - INT3 is added to QMoE version 1. There is no QMoE v2 registration and no Microsoft-domain available/released-version increase. - Models without an effective INT3 projection keep legacy shape/type propagation. Existing 2/4/8-bit byte semantics and attribute defaults are preserved, including expert_weight_bits=4, quant_type="int", and weights_prepacked=-1. - INT3 requires integer quantization, explicit weights_prepacked=0, activation_type="swiglu", fused interleaved gate/up, block sizes 32/64/128, block-aligned H/I, and matching FP16/BF16 activation/scale types. Omitting quant_type uses the existing "int" default; exporters should still emit it explicitly. - Support independent projection metadata, including (3,4), (3,3), and (4,3), and packed per-row block zero points. Infer H from activation shape and I from fused FC1 output rows, not FC2 packed bytes. Check shapes, types, symbolic dimensions, byte counts, and overflow. - Shared host-only ValidateInt3ProjectionContent requires block-aligned K (not N) at the projection level and checks exact buffer sizes, finite nonnegative FP16/BF16 scales, all-4 explicit zero points, canonical padding, and checked dimensions. Padded byte-count arithmetic is overflow-safe without changing the strict byte-aligned helper. - Native regressions verify that legacy INT4 models resolve to QMoE v1 and find the existing CPU kernel both with and without an explicit Microsoft-domain import. Adjacent Microsoft operators remain at v1. - Generated QMoE documentation is synchronized; non-QMoE documentation is unchanged. ## Validation of the v1 Revision Commit eda7353 was validated inside Docker jiafa-dev: - Offline Python reference tests: **223 passed, 1 skipped**. The historical optional external-checkout case was removed in 6c875ae; current results are listed below. ONNX 1.23.2, NumPy 2.4.6, ml_dtypes 0.6.0; BF16 coverage executed. - Actual current ORT schema, registration entry point and QMoE test sources compiled using cached Ninja-generated flags and linked with cached ORT libraries: **17 passed** (11 schema/registry tests and 6 content/byte-helper tests). - C++ filter: QMoEShapeInferenceTest.*:QMoEFormatValidationTest.*. New test files are collected by the existing contrib-test CMake glob. - Pinned clang-format 20.1.8, Ruff 0.12.12 check/format, editor diagnostics and git whitespace checks passed. - QMoE description was checked byte-for-byte using the documentation generator's rendering rules; all other operator documentation is byte-identical to the preceding PR head. - This is isolated current-source compilation/linking, **not a clean full ORT build** or CUDA inference qualification. The cached build predates two neighboring schemas, so their current definitions were also compiled for linking. Container CUDA runtime paths and system -lrt replaced unavailable cached paths. Only the existing MLAS GNU-stack linker warning remained. - Hosted CI for this new head is pending. Previous external-reference interoperability results are historical; that opt-in test was not rerun for this schema-only revision. ## Scope and Review Gates This PR adds **no INT3 execution kernel**. Existing v1 CPU/CUDA/WebGPU registrations remain; schema acceptance does not establish INT3 inference support. Provider validation/execution, packed CUDA decode, bounded prefill, export and model qualification belong to subsequent work. The end-to-end follow-up now builds on this v1 revision; see the coordinated Draft PRs below. The host content validator has **no runtime call sites yet**: it accepts caller-supplied host buffers and is directly unit-tested. Schema inference checks metadata, not weight contents; model loading does not automatically scan scales or zero points. Python-only tails/empty dimensions/FP32-scale fixtures do not broaden the block-aligned FP16/BF16 operator profile. The v1-compatible extension approach supersedes this PR's earlier v2/domain-version proposal. Operator/exporter owner review and format freeze remain required. No approved ONNX datatype, standard operator, production quantizer/exporter, checkpoint importer, accuracy or performance claim is made. ## Scoped Checkpoint Audit The local Olive audit at a9290ab9bf668cae5a02ad7473c8cb0f5476d35e found WeightQuantizer and pack_to_uint8 restricted to 2/4/8 bits, with packing_factor = 8 // bits; the inspected GPTQ pass imports that quantizer, and PrecisionBits has no three-bit option. These inspected paths do not establish a compatible native INT3 checkpoint source. External GPTQ implementations and GGUF Q3_K/IQ3 were not audited. Future importers must verify code mapping, bit order, tensor orientation, grouping, scales, zero points, permutations, gate/up interleaving and FC2 orientation rather than relabel unknown checkpoint bytes. Unrelated local files and submodule changes are excluded. ## Follow-up End-to-End Work The experimental end-to-end implementation now builds on this PR's compatible QMoE v1 contract. The work is split across three coordinated **Draft PRs**: - **ORT runtime: [microsoft#33519](microsoft#33519, stacked directly on this PR. Adds CUDA execution for `(FC1, FC2)=(3,3), (3,4), (4,3)` with FP16/BF16, direct packed extraction in both projections, routing and fused SwiGLU, strict validation, bounded activation tiling, tests, and experimental model results. - **Olive quantization: [microsoft/Olive#2749](microsoft/Olive#2749. Adds symmetric grouped INT3 RTN with offset-4 codes, stored-scale rounding, canonical row-local three-bit packing/unpacking, and ceil-bit storage shapes. - **Mobius export: [onnxruntime/mobius#783](onnxruntime/mobius#783. Enables expert-only INT3 overrides for the three width pairs while retaining the model-wide INT4 fallback, using existing native fused QMoE graph and external-data plumbing. Together these changes exercised source checkpoint -> Olive quantization -> Mobius ONNX export -> ORT CUDA inference. Six small cross-project cases cover both activation dtypes and all three width pairs. Full Qwen3-30B-A3B exports and model measurements cover all routed expert gate/up/down weights at INT3 `(3,3)` versus matched INT4 `(4,4)`, with 48 QMoE layers, block64, FP16, attention/embedding INT4 and head INT8. Graph-plus-weight size is 12,782,047,512 versus 16,405,926,354 bytes, a **22.09% reduction**. The runtime PR records A100 measurements at input lengths 128/512/2048/4096 and output length 32. **Current INT3 inference is substantially slower than INT4; this is not a performance delivery.** The paired 5-shot MMLU pipeline check uses only one question per subject (57 total): INT3 40/57 and INT4 49/57, with matching document/prompt/target hashes. These are **not full MMLU scores**; no completed full-MMLU result is claimed here. ### Delivery Plan Status These follow-ups do **not** complete planned PR2 through PR4: | Planned stage | Implemented evidence | Remaining gate | | --- | --- | --- | | PR2: CUDA correctness | Independent references, bounded execution, actual CUDA Session coverage | Clean provider-target build and formal integration-test acceptance | | PR3: packed decode | Direct packed FC1/FC2 execution for all three pairs | Versioned provider-private prepack, cache identity/lifetime and prepack-cost qualification | | PR4a: baseline prefill | Functional bounded direct-packed prefill, including full-model long prompts | Required selected-expert/tiled dequantization plus existing GEMM and dedicated acceptance tests | | PR4b/4c: optimized prefill | Not delivered | Winning kernel candidate, integration and performance acceptance | Before submission, the follow-up runtime's 82 actual CUDA Session cases passed; Olive's 176 scoped quantization tests and Mobius's 18 new tests also passed. Runtime experiments reused isolated cached build artifacts with explicit ABI alignment, **not a clean full build**. Formal OpTester execution, capture, concurrency, lifetime and full-model qualification remain open. Full-model mixed-pair qualification is also open; small fixtures do not close that gate. See microsoft#33519 and its committed results report/JSON for protocols, provenance, measurements and limitations. The checkpoint audit above describes the inspected pre-change Olive revision; microsoft#2749 supplies the subsequent producer changes. None of these follow-ups changes this PR's P0/P1-only diff or constitutes ONNX standard approval or production qualification. ## Review Fixes and Padding Experiment Follow-up Commit e45296b integrates the no-exceptions assertion guards, symbolic projection-row consistency fix and regression cases, and unpacked zero-point code clarification directly into this PR. Rebuilt current-source normal schema/format tests: **17/17 passed**; format tests compiled with ORT_NO_EXCEPTIONS plus the real helper implementation: **6/6 passed**. Pinned clang-format, whitespace and schema/doc synchronization checks passed. These remain isolated cached-dependency builds, not clean full or globally exception-disabled builds. Padding experiments remain separately reviewable in microsoft#33530: two corrected full A100/CUDA13 runs, 108 cases each, with all128expert rotation covered per full trial. For block64,24->32byte padding adds33.33% weight payload and is approximately11% slower for decode projection and10% slower for32token projection in the tested scalar-load implementation. This is not a whole-model or pure-bandwidth conclusion and does not rule out future vectorized private layouts. The padding design thread remains open for reviewer acceptance; experiment source and raw records are not added to this P0/P1 diff. ## Existing CI Integration Commit 8a41267 removes the dedicated packed-reference workflow and runs the suite from the existing Python quantization test stage in tools/ci_build/build.py, after unittest discovery when contrib ops are enabled. CMake copies the complete reference directory and golden_vectors.json into the test build output. Explicit pytest execution is necessary because unittest discovery does not collect these module-level pytest functions. Existing Python CI already installs pytest; no new workflow or dependency-install job is introduced. Validated by copying the directory with cmake -E copy_directory and executing the exact new pytest command extracted from build.py: **223 passed, 1 skipped** at 8a41267 (historical optional checkout test, removed in 6c875ae). Ruff check/format and git whitespace checks passed. Hosted CI for this head is not yet verified. ## Latest Review Fixes Commit 6c875ae addresses the latest review comments: - Removes the pinned external ONNX reference checkout test entirely. The suite is self-contained: **223 passed**, with no external-checkout skip. The ONNX submodule gitlink is unchanged by this PR. - Requires explicit `activation_type="swiglu"` for the INT3 fused profile and rejects the default/non-SwiGLU cases. Legacy INT4 activation behavior remains covered. Schema documentation is synchronized. - Removes the projection-level N-alignment restriction (K remains block-aligned), removes unreachable weight-tail validation, and computes each expected zero-point byte once before checking rows. Adds an N=127 acceptance regression. - Adds standard copyright/license headers and trailing newlines, restores the unrelated CPU model test to the main-branch version, and fixes CMake COMMAND indentation. Recompiled current affected C++ sources with cached dependency libraries: **18/18 normal schema/format tests passed** and **7/7 format tests passed with ORT_NO_EXCEPTIONS and the real helper implementation**. These are scoped cached-dependency builds, not a clean whole-ORT or globally no-exceptions build. The exact existing-CI pytest command also passed after CMake-copying the suite into a temporary build-output directory. Pinned clang-format, Ruff check/format, schema/doc synchronization and git whitespace checks passed. Hosted CI for this new head has not yet been verified. --------- Signed-off-by: David Fan <jiafa@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Companion exporter for microsoft/onnxruntime#33519 (runtime over foundation microsoft/onnxruntime#33394) and microsoft/Olive#2749.
Validation
18 newly added tests passed; touched-file Ruff passed. Prior extended exporter/reference run passed 170 tests. Actual Olive-to-Mobius-to-CUDA fixtures passed all three width combinations for FP16/BF16, and full Qwen3 all-expert INT3 export/execution was audited experimentally.
The ORT runtime companion is Draft and has no clean-build/production-qualification claim; INT3 is currently slower than INT4. See its recorded protocols and results. This producer change alone does not supply runtime INT3 support or complete delivery-plan PR2-PR4.