[WebGPU] Fuse rank-32 quantized LoRA with MatMulNBits by qjia7 · Pull Request #33370 · microsoft/onnxruntime

Jiajia Qin (qjia7) · 2026-10-14T12:18:00Z

Description

Adds com.microsoft.MatMulNBitsQLoRA fusion for the base projection, LoRA down/up projections, and residual addition. Supported raw MatMulNBits/Add patterns fuse automatically on WebGPU at extended graph optimization or higher. Adapters remain replaceable; base weights are unchanged.

Empty packed defaults let one raw model run on CPU and fuse on WebGPU without storing full-size zero weights. Fusion preserves initializer shapes and data: existing empty defaults stay empty, and full-size zero defaults retain their computation. General empty-weight MatMulNBits support is split into #33559 and remains included here until that PR merges.

The current WebGPU kernel targets FP16/FP32, INT4 base weights, rank-32 INT8 adapters, and block size 32. The operator definition allows other MatMulNBits quantization settings and arbitrary ranks.

Motivation

Reduce GPU dispatches and duplicated work in quantized LoRA inference while preserving runtime adapter switching.

Optimization method

Prefill: share activation quantization between base/down projections on eligible DP4A paths; fuse up/Add into the base output. Use subgroup-matrix acceleration where available.
Decode: compute base/down together where supported, followed by one up/Add dispatch; retain fallback paths.
Empty adapters: skip LoRA computation and temporaries in the fused WebGPU kernel.

Input upload batching was split into #33491 and has been merged. It is already included in both sides of the measurements below.

Performance

RTX 5080, built-in WebGPU/D3D12, FP16, accuracy level 4, batch 1, capture enabled, 128 generated tokens with readback. October 23 medians of ten requests (two reversed-order rounds, three warmups each). Original and converted/fused graphs use the same PR runtime (507021fd21). TTFT is time to first token.

Prompt tokens	Adapter	TTFT ms: unfused → optimized	Decode tok/s: unfused → optimized
128	Off	32.87 → 31.22	197.24 → 235.14
128	On	79.50 → 40.81	112.87 → 199.15
1024	Off	171.21 → 162.79	195.42 → 232.69
1024	On	221.92 → 173.21	112.44 → 196.35

Adapter-on TTFT falls 22–49% and decode improves approximately 75%; a second adapter shows similar gains. Adapter-off decode improves 19%. Capture-disabled runs also improve; session creation remains approximately 6.8 seconds.

In six sample prompts, adapter-off outputs match the original; with adapters active, 4/6 generated sequences match (logit RMSE 0.076). Broader quality validation remains necessary.

Follow-ups

Compose with QKV/MLP fusion to close the approximately 10% adapter-off decode gap against a pruned base-only graph.
Investigate the repeatable approximately 1.5 ms / 5% increase in 128-token TTFT on unchanged models versus main; decode is essentially unchanged.
Support more ranks/quantization settings and direct DequantizeLinear/MatMul adapter-pattern matching.

Adapter-on/off states require separate graph-capture identities; see onnxruntime-genai#2740.

Latest local validation: build and lint passed; 37 targeted tests passed, with one subgroup-matrix test skipped for device capability.

Skip uploader teardown before acquiring telemetry locks when Windows is detaching DLLs for process termination. WinHTTP callbacks and worker threads are already gone at that point, so draining them can wait forever. Keep normal object destruction and explicit DLL unload cleanup. Validated against captured native hang stacks and identical machine code at all resolved frames. A pending-uploader fault injection hangs the unchanged runtime and exits in 20 of 20 fixed trials. Both WebGPU modes completed 1024-prompt, 32-output-token native profiles with LoRA on and off; explicit unload also passed.

Add an opt-in MatMulNBitsLoRA operator and graph fusion for INT4/block-32 base weights with replaceable FP16 rank-32 adapters. Empty adapters execute only the base projection. Match original and offline-transposed layouts, absorb private adapter Transpose nodes, and preserve overridable inputs. Use two fusion implementations on the same transposed model: - fuse_down: base + T, followed by base + T * Bt^T. - fuse_up: T, followed by base with an integrated T * Bt^T epilogue. Default session.webgpu.lora_fusion_mode to auto: select fuse_up for M > 1 prefill and fuse_down for M == 1 decode on each invocation. Use X's per-matrix row dimension, independent of batch size. Retain explicit fuse_down/fuse_up overrides for comparisons. A one-token prompt uses the decode path. Mode selection applies to transposed adapters only. Both implementations read transposed B and T as vec4 FP16 components. Share the decode GEMV reduction and prefill down projection, preserving FP16 rounding, 128-thread decode producers and 64-thread standalone output. Keep DP4A activation quantization for the base, with original FP16 X for T. Name internal flags has_lora_down/has_lora_up by the fused projection. Retain compatible original-layout and base-kernel fallback paths. Validation on Windows/D3D12, RTX 5080, built-in and dynamic-plugin WebGPU: - 26 focused native tests pass with no skips; cover omitted mode, auto, both forced paths, decode, prefill, batches and empty/partial adapters. - 192 auto/default full-model cases and 4024 full-vocabulary logit vectors match the saved original unfused FP16 reference bitwise; rewind matches. - Cover adapter on/off and graph capture on/off for both providers. - Traces/shader dumps confirm 196 fused projections switching from prefill fuse_up to decode fuse_down, vec4 bindings, no adapter transpose, and base-only work with LoRA off. - All 72 native 1024-input/300-output benchmark sequences match exactly.

Add opt-in graph fusion and WebGPU kernels for INT4 base projections with block-32 INT8 adapters. Preserve replaceable packed weights, original INT8 arithmetic, partial overrides, and base-only execution for empty adapters. Use fuse_up for prefill and fuse_down for one-row NVIDIA decode, with packed vec4 loads. Add strict numerical and graph-matching tests and document native GenAI usage.

Mustapha Jaber (mustjab) · 2026-10-14T17:40:06Z

Hi Jiajia Qin (@qjia7), we tested this branch on an NVIDIA GeForce RTX 4070 (Windows D3D12, built-in WebGPU) and ran into two numerical tolerance issues in the unit test suites:

MatMulNBitsQLoRA.DecodeSubgroupReduction (onnxruntime/test/contrib_ops/matmul_nbits_qlora_test.cc:199):
- Fails at K=8192, N=64, acc_f32=0 where the FP16 inner-loop accumulation before subgroupAdd(f32(partial)) slightly exceeds the 0.002 absolute / 0.001 relative tolerance against the double-precision reference:
  - i:13: expected = 0.0883179, actual = 0.0906982, diff = 0.00238037 (tol = 0.00208832)
  - i:26: expected = 0.0225983, actual = 0.0246277, diff = 0.00202942 (tol = 0.00202260)
- Note: acc_f32=1 and smaller K (2048, 4096) pass, as does NarrowSmallMReduction and all 7 MatMulNBitsQLoRAFusion* optimizer tests.
Zero tolerance (0.0, 0.0) in MatMulNBitsLoRAFusion numerics tests (onnxruntime/test/optimizer/matmul_nbits_lora_fusion_test.cc):
- MatMulNBitsLoRAFusionTransposedNumerics, MatMulNBitsLoRAFusionTransposedProjectionNumerics, MatMulNBitsLoRAFusionTransposedBatchedNumerics, and MatMulNBitsLoRAFusionMatchesWebGpuDecodeAndPrefill pass 0.0, 0.0 tolerances to RunWebGpuFusionTransformerTest.
- On RTX 4070, the fused FP16 kernel exhibits 1–2 ULP rounding differences (e.g., diff = 6.10e-05 to 0.00390625) compared to the unfused MatMulNBits + MatMul + MatMul + Add graph due to reduction order differences.

…its-qlora

Retain FP16/FP32 MatMulNBitsQLoRA and remove floating-only schema, transformer, kernels and shader branches. Preserve QLoRA dispatch policy and packed adapter arithmetic. Validation: 38 native tests, 32 exact full-model comparisons, WASM syntax checks and lint. Isolated built-in WebGPU before/after timings show no repeatable material regression; record sanitized results.

…its-qlora

Accumulate the subgroup-matrix adapter delta independently and add it once to the base result to preserve cancellation. Reuse consumed input tiles for staging without adding dispatches, and retain the existing FP32 input support in the 8x8x8 path. When down weights are empty, evaluate an active up projection so dequantization overflow and NaN behavior match the unfused graph. Update the operator documentation and regression tests. Validated with the Vulkan Release build, all 31 QLoRA tests, and C++ formatting checks.

Explicitly initialize the 8x16x16 subgroup-matrix adapter accumulator on each output-column iteration. On Intel/Vulkan, implicit initialization retained the preceding group's sum and produced incorrect output values. Verified with the existing cancellation and narrow-reduction regressions, all 34 PR-added tests on built-in Vulkan, 89 WGSL template tests, and built-in/plugin Vulkan builds and model benchmarks.

Mustapha Jaber (mustjab) · 2026-10-31T00:26:53Z

Nit: In subgroup_matrix_matmul_nbits_8x16x16.wgsl.template (and tile_B in 8x8x8.wgsl.template), qlora_add reuses lora_a and tile_b as staging buffers across column-group iterations, so the reload loops should also zero-fill out-of-bounds global_m >= uniforms.M / global_n >= uniforms.N entries (like 16x16x16_128.wgsl.template does) instead of leaving prior staged values in padded rows/columns.

QLoRA addition reuses the adapter input tiles to stage intermediate results. Explicitly zero padded entries when reloading lora_a and tile_b in the 8x16x16 shader and tile_B in the 8x8x8 shader, matching the existing 16x16x16 handling. Verified with the built-in Vulkan build, 26 QLoRA/schema/dispatch tests (no skips), and 89 WGSL template tests. The 8x8x8 path was not executed on this Intel GPU.

Jiajia Qin (qjia7) · 2026-10-31T06:06:34Z

Nit: In subgroup_matrix_matmul_nbits_8x16x16.wgsl.template (and tile_B in 8x8x8.wgsl.template), qlora_add reuses lora_a and tile_b as staging buffers across column-group iterations, so the reload loops should also zero-fill out-of-bounds global_m >= uniforms.M / global_n >= uniforms.N entries (like 16x16x16_128.wgsl.template does) instead of leaving prior staged values in padded rows/columns.

Done

Mustapha Jaber (mustjab)

LGTM, thanks for addressing all the feedback!

Xiaofei Han (xiaofeihan1)

The staging capacity fix addresses my earlier concern. No further blocking findings.

### Description - Extend the WebGPU QKV and gated MLP graph fusions to `MatMulNBitsQLoRA` projections, producing `MatMulNBitsQLoRAQkv` and `MatMulNBitsQLoRAMlp`. - Reuse the existing fused base kernels when adapter-up weights are empty. Keep adapter weights as runtime inputs and evaluate active adapters through the QLoRA projection path. - Add operator schemas, generated operator documentation, and numerical tests for active and empty adapters, including skip normalization and QuickGelu patterns. ### Motivation and Context After #33370, QLoRA projections prevented the existing QKV and MLP fusions from matching. This left adapter-off Muffin decode slower than its pruned base-only AION graph. The new operators restore 28 QKV and 28 MLP fusions in the 28-layer model. ### Performance RTX 5080, built-in D3D12 WebGPU, graph capture on, batch 1, 16 prompt tokens and 128 generated tokens; median of four repetitions after two warmups, adapter off: | Model/path | Decode tokens/s | Decode ms/token | | --- | ---: | ---: | | Portable Muffin, QKV/MLP fusion disabled | 235.7 | 4.242 | | Portable Muffin, QKV/MLP fusion enabled | **259.8** | **3.849** | | Pruned AION base-only graph | 255.1 | 3.919 | | Direct QLoRA Muffin, fusion enabled | 260.2 | 3.843 | The fused portable model is about 10% faster than the disabled-fusion baseline and within about 2% of the pruned base-only graph. The direct QLoRA model gives the same result. ### Validation - Windows Release build: `onnxruntime_test_all` and `onnxruntime` passed. - Focused QLoRA MLP/QKV graph and WebGPU numerical tests: 4 test cases passed, covering active and empty adapters across the supported normalization and activation patterns. - Adapter-off full-model fixed-token logits matched AION exactly in a three-step check. FP16 fusion can change rounding relative to the unfused active-adapter graph. In one forced-token probe, a near tie changed the top token; the operator-level numerical tests passed.

Updated [Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.AspNetCore.Mvc.Testing's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Data.Sqlite's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime's releases](https://github.com/Microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime.Managed's releases](https://github.com/microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 18.11.0 to 18.12.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.NET.Test.Sdk's releases](https://github.com/microsoft/vstest/releases)._ ## 18.12.0 ## Security fix This release includes the fix for CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every version from 18.11.1 on is fixed, including 18.12.0. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Fix TestCaseStart/Stop events dropped for data-driven tests with shared TestCase.Id by @nohwnd in microsoft/vstest#16033 * Stop the discovery manager factory from shadowing discoveryCriteria by @nohwnd in microsoft/vstest#16391 * Gate the HTTP link checker on a fingerprint of the broken links by @nohwnd in microsoft/vstest#16413 * Quote boolean conditions in shipped Microsoft.TestPlatform.targets by @nohwnd in microsoft/vstest#16404 * Obsolete API hygiene: the compatibility-safe changes by @azat-msft in microsoft/vstest#16420 * Add xxHash128 test ids, available but not default by @azat-msft in microsoft/vstest#16378 * Isolate attachment manager test temporary files by @mmitche in microsoft/vstest#16463 * Test ids report logger by @azat-msft in microsoft/vstest#16443 * Isolate code coverage files in published output by @azat-msft in microsoft/vstest#16483 * Preserve OutputType for Android test projects by @jonathanpeppers in microsoft/vstest#16496 * Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft in microsoft/vstest#16462 * Mark the xxHash128 test id API as experimental by @azat-msft in microsoft/vstest#16465 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16633 * Fix localized resource paths in CLI source-build/non-Windows packages by @nohwnd in microsoft/vstest#16644 **Full Changelog**: microsoft/vstest@v18.11.0...v18.12.0 ## 18.11.1 ## Security fix This release fixes CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0. Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Security fix for CVE-2026-105544, see microsoft/vstest#16654 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16632 * Fix localized resource paths in CLI source-build packages by @nohwnd in microsoft/vstest#16643 **Full Changelog**: microsoft/vstest@v18.11.0...v18.11.1 Commits viewable in [compare view](microsoft/vstest@v18.11.0...v18.12.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Jiajia Qin (qjia7) and others added 10 commits October 12, 2026 11:53

Add Windows child-process coverage for 1DS teardown

625b47c

Optimize WebGPU QLoRA prefill with shared DP4A input and narrow tiles

2638ae2

Batch WebGPU input uploads to amortize adapter transfer overhead

8eab658

Optimize narrow WebGPU QLoRA projections with small-M DP4A

210584c

Optimize WebGPU QLoRA decode with register reuse and subgroup reduction

b3e1633

Document WebGPU LoRA performance and numerical limits

6a1cf7d

Merge main and fix WebGPU LoRA CI compatibility

7e70799

Merge remote-tracking branch 'origin/main' into feat/webgpu-matmul-nb…

5b64b23

…its-qlora

Jiajia Qin (qjia7) mentioned this pull request Oct 15, 2026

Fix graph capture identity after LoRA adapter activation microsoft/onnxruntime-genai#2740

Merged

Jiajia Qin (qjia7) added 7 commits October 15, 2026 15:05

Validate compact QLoRA capture and scope FP16 test tolerances

788de1b

Fix GCC range-loop warning in QLoRA decode test

8df7833

[WebGPU] Support FP16 and FP32 models in LoRA and QLoRA

c9d3692

[WebGPU] Fuse FP32 QLoRA decode base and down projection with DP4A

4e8a537

[WebGPU] Parallelize floating LoRA projections for prefill and decode

0243e40

Fix QLoRA fusion test template parsing on GCC 11

2ea9a83

Jiajia Qin (qjia7) changed the title ~~[WebGPU] Fuse rank-32 LoRA and QLoRA projections with MatMulNBits~~ [WebGPU] Fuse rank-32 quantized LoRA with MatMulNBits Oct 16, 2026

Jiajia Qin (qjia7) added 5 commits October 16, 2026 15:51

[WebGPU] Generalize K-parallel DP4A dispatch for narrow outputs

9ea931e

Merge branch 'main' into feat/webgpu-matmul-nbits-qlora

0cf697a

[WebGPU] Enable QLoRA fast paths by device capabilities

2ea6f9b

[WebGPU] Simplify QLoRA dispatch and reuse DP4A for short FP32 prefill

4103766

[WebGPU] Fix QLoRA CI tolerances and execute FP16 INT2 tests

51a970c

Jiajia Qin (qjia7) requested a balanced review from Copilot October 19, 2026 09:21

Copilot started reviewing on behalf of Jiajia Qin (qjia7) October 19, 2026 09:21

View session

Jiajia Qin (qjia7) added 2 commits October 19, 2026 17:25

[WebGPU] Note follow-up for arbitrary QLoRA ranks

2611096

Merge remote-tracking branch 'origin/main' into feat/webgpu-matmul-nb…

47214c4

…its-qlora

Jiajia Qin (qjia7) added 2 commits October 30, 2026 22:08

Jiajia Qin (qjia7) requested a review from Mustapha Jaber (mustjab) October 31, 2026 14:07

Xiaofei Han (xiaofeihan1) reviewed Nov 2, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/subgroup_matrix_matmul_nbits.cc Resolved

webgpu: Size QLoRA staging for four-wave BK=32 tiles

512827f

Mustapha Jaber (mustjab) approved these changes Nov 2, 2026

View reviewed changes

Xiaofei Han (xiaofeihan1) approved these changes Nov 3, 2026

View reviewed changes

Jiajia Qin (qjia7) enabled auto-merge (squash) November 3, 2026 04:27

Jiajia Qin (qjia7) merged commit e64a8ae into microsoft:main Nov 3, 2026

Jiajia Qin (qjia7) mentioned this pull request Nov 5, 2026

[WebGPU] Fuse QLoRA QKV and MLP projections #33901

Merged

BrewTestBot mentioned this pull request Nov 13, 2026

onnxruntime 1.32.0 Homebrew/homebrew-core#325304

Merged

1 task done

This was referenced Nov 14, 2026

Bump the nuget-minor-and-patch group with 2 updates vitorafgomes/scout#14

Closed

Bump the nuget-minor-and-patch group with 2 updates siebe41/SemanticRouter.NET#387

Open

Bump coverlet.collector and 25 others Cezart2332/ridelance-backend#30

Open

Navigation Menu

Repository navigation

[WebGPU] Fuse rank-32 quantized LoRA with MatMulNBits - #33370

[WebGPU] Fuse rank-32 quantized LoRA with MatMulNBits#33370
Jiajia Qin (qjia7) merged 49 commits into
microsoft:mainfrom
qjia7:feat/webgpu-matmul-nbits-qlora

Jiajia Qin (qjia7) commented Oct 14, 2026 •

edited

Loading

Uh oh!

Mustapha Jaber (mustjab) commented Oct 14, 2026

Uh oh!

Mustapha Jaber (mustjab) commented Oct 31, 2026

Uh oh!

Jiajia Qin (qjia7) commented Oct 31, 2026

Uh oh!

Uh oh!

Mustapha Jaber (mustjab) left a comment

Uh oh!

Xiaofei Han (xiaofeihan1) left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Jiajia Qin (qjia7) commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Optimization method

Performance

Follow-ups

Uh oh!

Mustapha Jaber (mustjab) commented Oct 14, 2026

Uh oh!

Mustapha Jaber (mustjab) commented Oct 31, 2026

Uh oh!

Jiajia Qin (qjia7) commented Oct 31, 2026

Uh oh!

Uh oh!

Mustapha Jaber (mustjab) left a comment

Choose a reason for hiding this comment

Uh oh!

Xiaofei Han (xiaofeihan1) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jiajia Qin (qjia7) commented Oct 14, 2026 •

edited

Loading