You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The last successful Mobius GPU fast-integration run (September 26) installed onnxruntime-gpu==1.30.0.dev20260812001 from the CUDA 12 feed. The opt-in QMoE CUDA tests in this PR are not part of the normal CI selections, so the CI checks do not qualify the merged ORT runtime for mixed-width QMoE. The CPU ORT GenAI E2E lane is separately pinned to ORT 1.29.0.
The separate experimental manual RTN recipe in Qwen3-30B-A3B V-only INT8-head RTN recipe for Mobius CUDA microsoft/olive-recipes#637 exported the full pinned Qwen3-30B-A3B checkpoint into an approximately 18 GB ONNX package with 24 (4,8) and 24 (4,4) QMoE layers. A bounded CUDA model-qa.py run returned 391 for 17 * 23 with ORT_QMOE_INT_DEQUANT_MAX_SCRATCH_BYTES=2147483648, and a 57-question MMLU smoke run completed. This used a locally built earlier ORT commit 50b8fcb, not the newly merged a11b4e5; it does not establish full-logit parity, production throughput, benchmark-scale quality, or validation of the final merged runtime.
ingest Olive fused expert checkpoints with FC1/FC2 INT2/INT4, INT4/INT8, INT8/INT4, or uniform INT8 weights under a model-wide INT4 fallback
allocate and validate FC1/FC2 packed weights, scales, and zero points independently, including adjacent layers with different expert widths
preserve legacy uniform INT4 QMoE emission and prevent mixed geometry from being incorrectly rewrite-fused
support authoritative Qwen3, Qwen3.5, and Qwen3.5-VL checkpoint prefixes
preserve pinned GGUF route fingerprints when the new QMoE source aliases are empty
Contract
Supported FC1/FC2 expert widths are (4,4), (2,4), (4,8), (8,4), and (8,8) with a model-wide INT4 fallback. Differing-width nodes emit projection-specific bit attributes, including FC3=FC1 for fused SwiGLU. Uniform INT8 nodes emit expert_weight_bits=8 with no projection-specific overrides. A model-wide INT8 fallback is not enabled.
Checkpoint preprocessing fails closed for missing, partial, stale, malformed, or unsupported expert sidecars. Plain and regex overrides follow Olive producer semantics.
Opt-in MOBIUS_QMOE_CUDA_TEST=1 python -m pytest tests/integration/qmoe_cuda_test.py -q: 2 passed on A100 with isolated ORT CUDA build at merged commit ee5f6e7. This exports a small Qwen3-MoE model using Olive-style fused sidecars, runs adjacent (4,8)/(8,4) and uniform (8,8) layers, checks full FP16 logits against an independent dequantized PyTorch reference (rtol=atol=0.01), and confirms each QMoE executes on CUDA via profiling. The full graph assigns shape/control nodes to CPU; it does not claim an all-CUDA graph.
Opt-in MOBIUS_QMOE_REAL_SLICE_CUDA_TEST=1 python -m pytest tests/integration/qmoe_cuda_test.py -k real_qwen3 -q: 2 passed on A100 using layer-0 experts 0–3 and sliced embedding/router weights from pinned Qwen3-30B-A3B. Olive RTN tensor producer created mixed (4,8)/(8,4) sidecars; both reloaded exports ran QMoE on CUDA and matched independently unpacked full logits at rtol=atol=0.01 (maximum absolute errors 0.005148/0.004774). Reduced four-expert top-2 routing and synthetic surrounding decoder are intentional; no full-checkpoint parity is claimed.
touched-file Ruff lint and formatting, python -m mypy src/mobius/_weight_utils.py, and git diff --check passed
Runtime boundary
microsoft/onnxruntime#32743 provides a bounded CUDA correctness fallback for (4,8) and (8,4) despite its earlier summary listing only 2/4-bit cases. It dequantizes expert weights into FP16/BF16 scratch and requires canonical raw weights, 3-D scales, block size 16–256, divisible reduction dimensions, and scratch within its configured cap. Uniform (8,8) uses the native packed INT8 CUDA path: the tested tiny model needed block size divisible by 32 and both FC reduction dimensions divisible by 64. The synthetic test uses small weights with zero attention to isolate routed experts. The separate real-slice test validates selected checkpoint experts via Olive RTN tensor production, not complete Olive pass orchestration or the original 128-expert top-8 model. Full-checkpoint parity, prefill/decode generation parity, throughput, and peak VRAM have not been validated. For Qwen3-30B-A3B, FP16 expert scratch alone is approximately 1.125 GiB per MoE layer, above the fallback default 1 GiB: configure ep.cuda.qmoe_int_dequant_max_scratch_bytes (or the runtime's corresponding environment override) and measure before claiming deployability at production size. CPU mixed-width execution is not qualified. microsoft/onnxruntime#32761 adds a packed decode subset for 2/4-bit combinations, not the (4,8) INT8 path.
Follow the ORT QMoE projection-specific bit-width contract for Olive checkpoints with INT2 gate/up experts and INT4 down experts. Preserve legacy uniform INT4 emission and add strict checkpoint, package, and rewrite regression coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🟡 Changes recommended
Mixed and legacy per-layer QMoE plans can diverge between graph construction and weight preprocessing, causing valid layer combinations or dense fallbacks to fail weight binding.
Get a fresh assessment by requesting another Copilot review.
Resolve every routed layer against its effective Olive layout so mixed and legacy uniform QMoE layers can coexist. Fail closed when packed sidecars target dense fallbacks and honor projection-specific packing geometry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Reject routed-expert exclusions and projection-specific GPTQ/AWQ layouts that cannot bind the current graph, while preserving redundant dense-fallback overrides. Validate every routed layer's Olive sidecars in mixed plans.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Two-layer QMoE test misses per-layer attributes and initializer bindings
src/mobius/models/moe_test.py:526
This two-layer regression validates only the preprocessed dictionary, so the new per-layer qmoe_source_paths graph wiring could assign mixed-width attributes to the wrong layer while this test still passes. The linked acceptance criterion also requires the multi-layer case to inspect node attributes and final initializer bindings. Please build the two-layer package, associate each QMoE node by its FC1 initializer input, assert that only mixed_layer has the FC-specific bit attributes, apply the processed weights, and compare both layers' bound initializer bytes.
Strengthen the two-layer regression to associate each QMoE node with its FC1 initializer, validate projection-specific bit attributes on only the mixed layer, and compare both layers' bound FC1 and FC2 initializer bytes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Addressed the previously missed multi-layer coverage in 5fdadb6. The two-layer regression now associates each QMoE node with its layer-specific FC1 initializer through the emitted producer chain, asserts that only the selected mixed layer carries fc1=2/fc2=4/fc3=2 attributes, applies the processed weights, and byte-compares both layers’ bound FC1/FC2 initializers. The focused QMoE suite passes (156 tests).
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
GPTQ/AWQ parent overrides are resolved using incompatible Olive matching semantics.
Review effort: Balanced Findings: None
Previously missed (1)
In code that hasn't changed since last review
Use generic subtree matching for GPTQ and AWQ overrides
src/mobius/_weight_utils.py:207
This applies Olive's literal override/substring-exclusion rules to GPTQ and AWQ too. Those formats use QuantizationConfig.for_module subtree matching, so a valid parent override such as model.layers.0.mlp is currently ignored for both expert projections; the graph can retain the global geometry and later mismatch the packed checkpoint instead of failing closed. Delegate non-Olive resolution to the generic matcher before applying Olive-specific semantics.
Delegate GPTQ and AWQ module rules to the generic subtree matcher while preserving Olive's literal override and substring exclusion semantics. Add parent, near-match, and exclusion regressions and document the producer-specific contract.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Addressed review 5284088183 in a0a05d0. Non-Olive QMoE resolution now delegates to QuantizationConfig.for_module, restoring generic plain-path subtree matching for GPTQ/AWQ while leaving Olive literal overrides and substring exclusions unchanged. Added GPTQ/AWQ regressions for a matching parent override, a substring-only near miss, and a parent exclusion; updated the design contract. Validation: 162 QMoE tests passed, mypy and lint passed.
ORT's merged correctness fallback covers INT4/INT8 and INT8/INT4 even though its PR summary only lists INT2 combinations. Distinguish upstream CUDA coverage from unverified Mobius fused-SwiGLU end-to-end execution and CPU support.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Exercise Olive-style fused Qwen3-MoE expert sidecars in adjacent INT4/INT8 layers and a uniform INT8 layer using a pinned post-fallback ORT wheel. Compare full logits with an independently dequantized reference and prove QMoE nodes execute on CUDA. Document the INT8 packed-kernel alignment and large-model scratch limits.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
Quantize one reduced Qwen3-30B-A3B expert layer using Olive's RTN tensor producer, serialize its sidecars, and verify mixed INT4/INT8 exports against an independent dequantized reference on pinned ORT CUDA. Keep the integration opt-in and document its deliberate limits: reduced routing, synthetic surrounding decoder, and no full-model performance or quality claim.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: titaiwang <titaiwang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status update (September 28, 2026)
ee5f6e7on September 24, and Add packed INT2 and mixed-width CUDA QMoE decode microsoft/onnxruntime#32761 merged asa11b4e5on September 28. The latter optimizes packed decode for(2,2),(2,4), and(4,2);(4,8)/(8,4)still use the bounded dense correctness fallback. These merges do not automatically update Mobius CI's installed ORT wheel.onnxruntime-gpu==1.30.0.dev20260812001from the CUDA 12 feed. The opt-in QMoE CUDA tests in this PR are not part of the normal CI selections, so the CI checks do not qualify the merged ORT runtime for mixed-width QMoE. The CPU ORT GenAI E2E lane is separately pinned to ORT 1.29.0.(4,8)and 24(4,4)QMoE layers. A bounded CUDAmodel-qa.pyrun returned391for17 * 23withORT_QMOE_INT_DEQUANT_MAX_SCRATCH_BYTES=2147483648, and a 57-question MMLU smoke run completed. This used a locally built earlier ORT commit50b8fcb, not the newly mergeda11b4e5; it does not establish full-logit parity, production throughput, benchmark-scale quality, or validation of the final merged runtime.Summary
Contract
Supported FC1/FC2 expert widths are
(4,4),(2,4),(4,8),(8,4), and(8,8)with a model-wide INT4 fallback. Differing-width nodes emit projection-specific bit attributes, including FC3=FC1 for fused SwiGLU. Uniform INT8 nodes emitexpert_weight_bits=8with no projection-specific overrides. A model-wide INT8 fallback is not enabled.Checkpoint preprocessing fails closed for missing, partial, stale, malformed, or unsupported expert sidecars. Plain and regex overrides follow Olive producer semantics.
Validation
python -m pytest src/mobius/models/moe_test.py src/mobius/components/_moe_test.py src/mobius/models/qwen35_test.py src/mobius/integrations/gguf/_builder_core_test.py src/mobius/integrations/gguf/_moe_cohort_test.py -q --tb=short— 399 passedpython -m pytest src/mobius/rewrite_rules/_qmoe_fusion_test.py src/mobius/_weight_utils_test.py -q --tb=short— 172 passedMOBIUS_QMOE_CUDA_TEST=1 python -m pytest tests/integration/qmoe_cuda_test.py -q: 2 passed on A100 with isolated ORT CUDA build at merged commitee5f6e7. This exports a small Qwen3-MoE model using Olive-style fused sidecars, runs adjacent (4,8)/(8,4) and uniform (8,8) layers, checks full FP16 logits against an independent dequantized PyTorch reference (rtol=atol=0.01), and confirms each QMoE executes on CUDA via profiling. The full graph assigns shape/control nodes to CPU; it does not claim an all-CUDA graph.MOBIUS_QMOE_REAL_SLICE_CUDA_TEST=1 python -m pytest tests/integration/qmoe_cuda_test.py -k real_qwen3 -q: 2 passed on A100 using layer-0 experts 0–3 and sliced embedding/router weights from pinned Qwen3-30B-A3B. Olive RTN tensor producer created mixed (4,8)/(8,4) sidecars; both reloaded exports ran QMoE on CUDA and matched independently unpacked full logits atrtol=atol=0.01(maximum absolute errors 0.005148/0.004774). Reduced four-expert top-2 routing and synthetic surrounding decoder are intentional; no full-checkpoint parity is claimed.python -m mypy src/mobius/_weight_utils.py, andgit diff --checkpassedRuntime boundary
microsoft/onnxruntime#32743 provides a bounded CUDA correctness fallback for
(4,8)and(8,4)despite its earlier summary listing only 2/4-bit cases. It dequantizes expert weights into FP16/BF16 scratch and requires canonical raw weights, 3-D scales, block size 16–256, divisible reduction dimensions, and scratch within its configured cap. Uniform (8,8) uses the native packed INT8 CUDA path: the tested tiny model needed block size divisible by 32 and both FC reduction dimensions divisible by 64. The synthetic test uses small weights with zero attention to isolate routed experts. The separate real-slice test validates selected checkpoint experts via Olive RTN tensor production, not complete Olive pass orchestration or the original 128-expert top-8 model. Full-checkpoint parity, prefill/decode generation parity, throughput, and peak VRAM have not been validated. For Qwen3-30B-A3B, FP16 expert scratch alone is approximately 1.125 GiB per MoE layer, above the fallback default 1 GiB: configureep.cuda.qmoe_int_dequant_max_scratch_bytes(or the runtime's corresponding environment override) and measure before claiming deployability at production size. CPU mixed-width execution is not qualified. microsoft/onnxruntime#32761 adds a packed decode subset for 2/4-bit combinations, not the(4,8)INT8 path.Addresses #735.