[CUDA] Skip padded GQA work on CPU and CUDA by tianleiwu · Pull Request #33832 · microsoft/onnxruntime

Tianlei Wu (tianleiwu) · 2026-11-03T06:04:28Z

Description

Support batched right-padded GQA chunks and populated idle entries (query_seqlens[b] = 0) on CPU and CUDA, including single-token steps. Skip padded preprocessing and attention work while preserving physical tensor strides and the sliding-window history needed by the first valid query in each chunk.

This follows #33503 and #33505, both already merged. Other execution providers continue to reject query_seqlens; the legacy multi-token/past-context batch restriction remains when the input is absent.

Summary of Changes

Length semantics and correctness

Accept 0 <= Q[b] <= min(S, T[b]) with T[b] >= 1. An idle entry keeps its absolute total and cache unchanged. Initially empty idle entries remain unsupported.
Honor explicit lengths at S = 1, rather than treating every entry as active.
Fix CPU flash dispatch for equal resident totals but different past lengths: the causal offset cannot be shared in that case.
Preserve staging/eviction behavior: eviction must not discard the preceding W - 1 positions needed by the first token of a chunk.
Clarify that total_sequence_length bounds absolute totals, not windowed cache capacity.

CPU

Bound unpack/transpose, RoPE, FP16 conversion, naive QK/softmax/SV and MLAS flash query work by the valid row count.
Keep padded physical strides for head, batch, output and bias addressing.
Skip idle attention while preserving required past-cache copies for separately allocated outputs.
Cover FP32/FP16 inputs and INT8/INT4 caches; retain default MLAS behavior when valid query counts are not supplied.

CUDA

Guard padding before Q/K/V loads, QK-Norm, RoPE, quantization and cache append.
Skip idle aliased-cache staging seed/copy-back and unused ungroup/dequantization rows.
Bound FlashAttention/MEA query tiles and output stores by device-side valid lengths.
Skip padded unfused softmax/output rows and use masked V scratch to avoid 0 * NaN contamination from unused cache rows. Include that scratch in runtime and bounded workspace calculations.
Bypass XQA, fast Flash decode and cuDNN SDPA with explicit query lengths because those paths do not represent idle query entries. No host synchronization is added to inspect query lengths.

Remaining work: unfused CUDA cuBLAS QK/SV GEMMs still use dense physical query dimensions. Partially filled fused tiles retain tile-granular arithmetic. Scratch allocations and upstream Q/K/V projection nodes are not made ragged by this change.

Regression coverage and documentation

Replay prompts [513, 1025] with S = W = 512, CPU cache capacity 1023 and CUDA capacity 512:

Chunk Valid queries Absolute totals

1 [512, 512] [512, 512]

2 [1, 512] [513, 1024]

3 [0, 1] [513, 1025]
Compare valid outputs and resident caches with independent unpadded single-entry full-cache calls. Include all-idle chunks, idle decode, resumed decode, mixed first/continuation entries, additional eviction, and equal-total/different-past cases.
Poison padded Q/K/V and unused floating-cache rows with NaNs. Prebind sentinel outputs and verify padded rows are not written. The sentinel check is an optimization regression, not a new public padding-output guarantee.
Exercise packed/unpacked inputs, RoPE, FP16/BF16 CUDA, FP32/FP16 CPU, and INT8/INT4 caches.
Assert native FlashAttention/MEA/math dispatch without CPU fallback, exercise MEA large-head accumulation, and replay CUDA graphs while changing device query lengths.
Update operator schema, generated operator documentation and CPU/CUDA guides.

Testing

Validated locally on H200 with CUDA 13 using the existing incremental build and feature-source overlays:

Rebuilt Python runtime, CPU/MLAS, CUDA provider and provider-test targets, including dependent FlashAttention/MEA translation units. Verified compiled feature-source and staged-provider parity.
GQA C++ suite: 123 passed, 14 skipped (137 tests).
Focused MEA/unfused workspace GTests: 23 passed. Both changed workspace test files compile; additional standalone layout/bounds checks passed. The main build has CUDA EP internal tests disabled, so the complete estimator/internal-test suite was not run.
Targeted Python query-length/windowed suites: 112 passed, 2 skipped, plus 148 passing subtests.
Final new optimization suite rerun: 36 passed, plus 48 passing subtests, including all three CUDA-graph backends.
Independent NumPy-oracle replay: 96 CPU calls and 156 CUDA calls passed; CUDA dispatch verified for FlashAttention, MEA, math and automatic selection.
All configured lintrunner checks and git diff --check passed.

The CPU output-sentinel test was also run before optimization and reproduced padded output writes for both flash settings and both cache-alias modes.

Checklist

CPU and CUDA changes included
Regression tests added/updated
Existing no-input behavior and other-EP rejection retained
Documentation and backend limitations updated

Enable batched and idle explicit query lengths while preserving sliding-window history. Bound CPU/CUDA preprocessing and supported attention paths by valid queries, fix CPU flash causal offsets, and add replay, padding-work, graph-capture and workspace coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

The regenerated GQA reference drops all 17 input and 4 output descriptions, omitting tensor shapes, cache layout, and the updated query_seqlens contract despite those descriptions remaining in the source schema.

1 open issue

Regeneration drops GQA parameter descriptions from published reference · New

What changed in this PR

Extends ONNX Runtime’s CPU and CUDA GroupQueryAttention support from #33503 and #33505 to batched right-padded chunks and populated idle entries, including single-token steps.

Changes:

Honors zero valid-query lengths while preserving physical tensor strides, cache contents, and sliding-window history.
Skips padded preprocessing and attention work, updates CUDA backend selection, and accounts for masked-value scratch space.
Adds regression coverage for idle entries, quantized caches, backend dispatch, and CUDA graph replay; updates the operator contract and guides.

File	Description
`onnxruntime/test/python/transformers/test_gqa.py`	Adds batched padding, idle-cache, and CUDA graph regressions.
`onnxruntime/test/providers/cuda/test_cases/group_query_attention_workspace_mea_unfused_test.cc`	Tests masked-value scratch layout and validation.
`onnxruntime/test/providers/cuda/test_cases/group_query_attention_workspace_estimate_test.cc`	Checks scratch inclusion in workspace estimates.
`onnxruntime/test/contrib_ops/group_query_attention_op_test.cc`	Adds batched and idle-entry cases; updates validation expectations.
`onnxruntime/core/mlas/lib/flashattn_qkv.cpp`	Limits quantized attention to valid query rows.
`onnxruntime/core/mlas/lib/flashattn_gqa.cpp`	Limits floating-point attention to valid query rows.
`onnxruntime/core/mlas/inc/mlas.h`	Adds optional valid-query length to GQA arguments.
`onnxruntime/core/mlas/inc/mlas_qkv_quant.h`	Adds valid-query length to quantized attention arguments.
`onnxruntime/core/graph/contrib_ops/bert_defs.cc`	Documents batched chunks, idle entries, and absolute-total bounds.
`onnxruntime/contrib_ops/cuda/bert/unfused_attention.h`	Adds query lengths and a distinct value-buffer stride.
`onnxruntime/contrib_ops/cuda/bert/unfused_attention.cu`	Skips padded softmax work and supports value scratch strides.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention.cc`	Wires explicit lengths into dispatch, staging, and scratch allocation.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_workspace.h`	Extends workspace structures for masked-value scratch.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_workspace_mea_unfused.cc`	Sizes and validates masked-value scratch.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_workspace_estimate.cc`	Detects explicit query lengths during estimation.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_workspace_bounds.h`	Records query-length presence in workspace bounds.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_workspace_bounds.cc`	Propagates query-length presence into workspace recipes.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_qkv.cuh`	Guards padded rows before preprocessing and cache writes.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_qdq.cuh`	Skips idle entries and unused dequantization rows.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.h`	Extends ungrouping and cache-copy interfaces with length guards.
`onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu`	Implements length-aware attention preparation and output handling.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/flash.h`	Adds fixed-stride valid-query lengths to Flash parameters.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_launch_template.h`	Disables full-tile assumptions for explicit query lengths.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_kernel.h`	Guards padded rows during split-result combination.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_api.h`	Extends the Flash cache API with query lengths.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_api.cc`	Passes query lengths into Flash kernel parameters.
`onnxruntime/contrib_ops/cuda/bert/flash_attention/block_info.h`	Derives effective query lengths without changing strides.
`onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/memory_efficient_attention.h`	Adds valid-query lengths to memory-efficient attention parameters.
`onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h`	Adds the kernel-side query-length pointer.
`onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/fmha_launch_template.h`	Bounds query work while preserving physical output strides.
`onnxruntime/contrib_ops/cuda/bert/attention_data.h`	Carries device query lengths through CUDA GQA.
`onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc`	Bounds transpose, rotary embedding, and attention by valid rows.
`onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h`	Allows batched chunks and zero valid-query lengths.
`onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h`	Trims attention work and handles differing causal offsets.
`docs/ContribOperators.md`	Updates the generated GQA contract.
`docs/contrib_ops/cuda/gqa.md`	Documents CUDA behavior, optimizations, and backend limits.
`docs/contrib_ops/cpu/gqa.md`	Documents CPU idle entries and sliding-window examples.

_{Review effort: Balanced}

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>

Ti-Tai Wang (titaiwangms) · 2026-11-04T21:29:44Z

Full review: changes recommended

Reviewed 22c72f5 → 8c632c8 (37 files) with Readability, Code, Critical, Deep, and Integration reviewers. Actual reviewer models were verified. One confirmed Major finding; no additional core implementation defect was established.

Findings ledger

ID	Severity	Raised by	Finding	Disposition
F1	Major	Lead, Code Reviewer	New CUDA tests lack a PyTorch CUDA-availability guard and fail Windows CI	Confirmed

F1 — Skip CUDA tests when PyTorch has no CUDA support

Locations: test_gqa.py:5560–5561, 5602–5603.

has_sdpa_kernel("MATH") returns True before checking torch.cuda.is_available(). An ORT build exposing the CUDA EP with CPU-only PyTorch therefore enters these new tests and fails when allocating CUDA tensors, without exercising attention.

This is already demonstrated by the Windows CUDA test job at the reviewed head: five new MATH cases fail with AssertionError: Torch not compiled with CUDA enabled—the CUDA-graph MATH case and all four windowed-idle MATH cases.

Minimal fix: Add a torch.cuda.is_available() skip guard to both CUDA test methods, or move that check ahead of the MATH return in the shared helper after checking its callers. Do not skip the entire class, since it also contains CPU tests. Confidence: high.

Non-blocking open questions

Critical Reviewer: The new H512 MEA test's shared-memory feasibility on SM86 was not verified. There is insufficient evidence to assert a failure; a shape-specific hardware run would resolve this.
Deep Reviewer: The MEA single-query head-folding path with S=1, Q=1, and explicit query lengths looked consistent statically, but that exact scenario was not executed in this review.

Coverage, dissent, and residual risk

Static tracing did not establish additional introduced defects in CPU/CUDA causal offsets, idle-cache preservation, physical strides, Flash split-combine, masked-V scratch, or runtime/estimator workspace consistency. The intentional fast-path bypass and padding-output sentinel checks are documented design choices, not additional findings. No Major/Critical finding was rejected or deferred.

git diff --check passed. No local builds, kernel execution, sanitizers, or performance benchmarks were run; this is not a runtime safety clearance. Specialized kernel variants and transitive consumers were not exhaustively audited. Other failing checks showed WebGPU operator-documentation differences and Web detached-ArrayBuffer errors; a causal relationship to this PR was not established, so they are not reported as introduced defects.

Check PyTorch CUDA support before advertising the math backend so CUDA-only regressions skip with CPU-only PyTorch without disabling CPU coverage. Keep math support on pre-SM80 devices and cover both availability cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tianlei Wu (tianleiwu) · 2026-11-04T23:56:37Z

Addressed F1 in 5cb1d43.

has_sdpa_kernel() now checks PyTorch CUDA availability before the MATH fast return. CUDA-only tests therefore skip with CPU-only PyTorch without disabling CPU coverage in the mixed class. MATH remains available on pre-SM80 CUDA devices. Added regressions for all three backend probes without PyTorch CUDA and for pre-SM80 MATH support.

Validation with the rebuilt PR runtime:

40 tests + 48 subtests passed, with no skips, on H200/SM90: CPU/CUDA padded chunks, native backend dispatch, CUDA graphs, and H512 MEA.
Simulated unavailable PyTorch CUDA with the ORT CUDA EP present: 12 CPU tests passed; all 24 CUDA tests skipped, including the five previously failing MATH cases. CUDA capability inspection and backend probing were forbidden during this run.
Additional forced-MEA check: S=1, Q=1, explicit query lengths, global attention, single KV head passed output/cache parity and native-dispatch assertions.
Ruff lint/format and whitespace checks passed.

The non-blocking SM86 H512 shared-memory question remains unverified because only SM90 hardware is available; the H200 result is not a portability claim.

Ti-Tai Wang (titaiwangms)

Reviewed the update through 5cb1d43. The previously reported Major finding (F1: missing PyTorch CUDA-availability guard) is resolved, with focused regression tests added. The Windows CUDA test job that previously failed now passes. No remaining blocking code findings were established by the full static review and focused follow-up. Approval is for the reviewed code, not a declaration that all merge checks are green: documentation generation/validation, React Native Android, and Linux training checks still show failures whose latest causes were not verified in this follow-up. No local builds, kernel execution, sanitizers, or benchmarks were performed; the previously noted non-blocking hardware-validation questions remain.

…chunks Support (openvinotoolkit#39495) ## Description Implements the optional `query_seqlens` input (index 16) of `com.microsoft.GroupQueryAttention`, added by ONNX Runtime in microsoft/onnxruntime#33505 (CPU) on top of microsoft/onnxruntime#33503 (schema, CUDA). The input gives the number of valid rows of a right-padded query chunk, so static-shape chunked prefill can pad its last chunk to the fixed chunk length. Before this change the ONNX frontend rejected any GroupQueryAttention node with 17 inputs. With `S` the padded chunk length, `T = seqlens_k + 1` and `Q = query_seqlens`, the past length becomes `P = T - Q` instead of `T - S`: query row `s` sits at absolute position `P + s` for the causal mask, the sliding window and the implicit rotary positions, and only the `Q` valid rows are new tokens. Following microsoft/onnxruntime#33832, batched chunks are supported, the input is honored at `sequence_length == 1`, and `Q = 0` marks an idle entry whose cache is unchanged. Models without the input are unaffected. ONNX Runtime references: - **Spec:** https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.GroupQueryAttention - **CPU kernel:** https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc - **CPU attention / cache write:** https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h - **Input checks:** https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h ## Approach - **Op:** `ov::op::internal::GroupQueryAttention` gets a `QUERY_SEQLENS` input with `has_query_seqlens()`. Validation matches ONNX Runtime: rank 1 with `batch_size` elements, past required (a typed-empty rank-4 Constant counts as past layout, matching an empty Parameter), `causal=1`, no shared KV, key/value length equal to the query's. - **ONNX FE:** accepts 17 inputs and forwards `query_seqlens` only when it is wired, so existing models keep a 16-input op and their IR is unchanged. - **Decomposition:** when `has_query_seqlens()`, `P = T - Q` (per batch entry) drives the mask, the KV write offset and the implicit rotary positions. The KV write keeps its static S-row shape at `[P, P + S)`: the cache stays left-aligned with every valid row in `[0, T)`, and the padded tail lands at or beyond `T`, where no valid query reads it and the next step overwrites it (ONNX Runtime requires `P + S <= capacity`). Writing only the `Q` valid rows would make the update shape data-dependent, which static-shape devices cannot compile. The public present buffer is separately sized to ONNX Runtime's `max(past_rows, total_sequence_length)` capacity contract (`group_query_attention_helper.h`), since `total_sequence_length` can pre-allocate headroom past this step's write; attention itself still runs over the tight `[P, P + S)` buffer. The windowed-cache staging path writes back only the valid rows, so eviction follows `T`. An idle entry (`Q = 0`, `P = T`) writes only at or beyond `T`, so its valid cache is unchanged; an idle windowed decode takes the staging path for the same reason. Padded K/V rows are zeroed with one `Select` each, since masking alone lets NaN filler reach valid rows through the scores and `0 * V` (ONNX Runtime CPU/CUDA never read them); single-token steps skip it, as their padded entries are idle. Padded rotary rows are capped at `seqlens_k` with one `Minimum`, which keeps them in range; their outputs are unspecified, as in ONNX Runtime. Graphs without the input lower exactly as before. - **GPU:** `StatelessKV` gets `P + S` as its present length, so the S rows land at `P` and SDPA's lower-right causal alignment gives row `s` the keys `<= P + s`, with no extra mask or copy. A windowed cache with a padded chunk uses the generic staging path, because `StatelessKV` evicts by its present length; decode keeps the `StatelessKV` sliding-window path. ## Scope In scope: CPU (common decomposition) and GPU for full-length, static-capacity and windowed caches, implicit and explicit rotary positions, float and int8-quantized KV, single and batched graphs. Out of scope: NPU compilers, which do not handle the input yet; `StatelessKV` honoring `total_sequence_length` headroom beyond `P + S` on GPU, and keeping a padded windowed chunk on the GPU `StatelessKV` sliding-window path (both planned as a follow-up). ## Validation - type_prop tests for the new input, batched and single-token acceptance, and every rejection. - ONNX FE tests with expected values from the ONNX Runtime CPU kernel: static cache with rotary, growing cache (batched path), windowed cache, single-token step with the input wired, explicit `position_ids`, int8 quantized cache, plus the microsoft/onnxruntime#33832 cases: batched chunk with an idle entry and NaN-poisoned padding, equal totals with different pasts, idle single-token step, windowed idle and active single-token steps. Padded output rows and cache rows at or beyond the valid length are unspecified, so the models compare only the valid region. The GPU run of the static-capacity cases is skipped because a static-capacity present output consumed by a non-Result op fails at runtime on GPU, which also happens without this change. - A growing-cache regression evaluates the unsliced present output directly against `total_sequence_length` (not `P + S`), checking both the resulting shape and that the write window and unwritten past prefix land correctly, matching the ONNX Runtime CPU/CUDA capacity contract (`group_query_attention_helper.h`). - A padded prefill chunk with 32 heads, 8 KV heads, head size 128, `S = 512`, `Q = 300` and `P = 1000` matches the ONNX Runtime CPU kernel on CPU (output max abs diff 1.2e-6, present KV bit-exact) for a static and a windowed cache, and its latency with the input matches the same chunk without it. ### FYI - microsoft/onnxruntime#33832 (merged after this PR was opened) extended the `query_seqlens` contract on CPU/CUDA: batched chunks, `Q = 0` idle entries and honoring the input at `sequence_length == 1`. This PR follows that contract; expected values come from the ONNX Runtime CPU kernel at that commit. ### Tickets: - CVS-199056 ### AI Assistance: - AI assistance used: yes - AI helped analyze, implement, test the optimal changes

Updated [Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.AspNetCore.Mvc.Testing's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Data.Sqlite's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime's releases](https://github.com/Microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime.Managed's releases](https://github.com/microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 18.11.0 to 18.12.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.NET.Test.Sdk's releases](https://github.com/microsoft/vstest/releases)._ ## 18.12.0 ## Security fix This release includes the fix for CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every version from 18.11.1 on is fixed, including 18.12.0. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Fix TestCaseStart/Stop events dropped for data-driven tests with shared TestCase.Id by @nohwnd in microsoft/vstest#16033 * Stop the discovery manager factory from shadowing discoveryCriteria by @nohwnd in microsoft/vstest#16391 * Gate the HTTP link checker on a fingerprint of the broken links by @nohwnd in microsoft/vstest#16413 * Quote boolean conditions in shipped Microsoft.TestPlatform.targets by @nohwnd in microsoft/vstest#16404 * Obsolete API hygiene: the compatibility-safe changes by @azat-msft in microsoft/vstest#16420 * Add xxHash128 test ids, available but not default by @azat-msft in microsoft/vstest#16378 * Isolate attachment manager test temporary files by @mmitche in microsoft/vstest#16463 * Test ids report logger by @azat-msft in microsoft/vstest#16443 * Isolate code coverage files in published output by @azat-msft in microsoft/vstest#16483 * Preserve OutputType for Android test projects by @jonathanpeppers in microsoft/vstest#16496 * Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft in microsoft/vstest#16462 * Mark the xxHash128 test id API as experimental by @azat-msft in microsoft/vstest#16465 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16633 * Fix localized resource paths in CLI source-build/non-Windows packages by @nohwnd in microsoft/vstest#16644 **Full Changelog**: microsoft/vstest@v18.11.0...v18.12.0 ## 18.11.1 ## Security fix This release fixes CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0. Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Security fix for CVE-2026-105544, see microsoft/vstest#16654 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16632 * Fix localized resource paths in CLI source-build packages by @nohwnd in microsoft/vstest#16643 **Full Changelog**: microsoft/vstest@v18.11.0...v18.11.1 Commits viewable in [compare view](microsoft/vstest@v18.11.0...v18.12.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings November 3, 2026 06:04

Copilot started reviewing on behalf of Tianlei Wu (tianleiwu) November 3, 2026 06:05

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Comment thread docs/ContribOperators.md Resolved

Copilot started work on behalf of Tianlei Wu (tianleiwu) November 3, 2026 09:11

View session

Restore GQA parameter descriptions

8c632c8

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>

Copilot finished work on behalf of Tianlei Wu (tianleiwu) November 3, 2026 09:14

Ti-Tai Wang (titaiwangms) self-requested a review November 4, 2026 21:00

Ti-Tai Wang (titaiwangms) approved these changes Nov 5, 2026

View reviewed changes

Tianlei Wu (tianleiwu) merged commit 527971b into main Nov 5, 2026

Tianlei Wu (tianleiwu) deleted the tlwu/gqa_batched_idle_skip_padding branch November 5, 2026 05:37

Ankit Maheshkar (ankitm3k) mentioned this pull request Nov 5, 2026

[Core][ONNX FE][GPU] GroupQueryAttention: query_seqlens right-padded chunks Support openvinotoolkit/openvino#39495

Merged

BrewTestBot mentioned this pull request Nov 13, 2026

onnxruntime 1.32.0 Homebrew/homebrew-core#325304

Merged

1 task done

Selena Yang (selenayang888) mentioned this pull request Nov 13, 2026

Skip inactive XQA split-KV CTAs using runtime sequence length #34214

Draft

This was referenced Nov 14, 2026

Bump the nuget-minor-and-patch group with 2 updates vitorafgomes/scout#14

Closed

Bump the nuget-minor-and-patch group with 2 updates siebe41/SemanticRouter.NET#387

Open

Bump coverlet.collector and 25 others Cezart2332/ridelance-backend#30

Open

Navigation Menu

Repository navigation

[CUDA] Skip padded GQA work on CPU and CUDA - #33832

[CUDA] Skip padded GQA work on CPU and CUDA#33832
Tianlei Wu (tianleiwu) merged 3 commits into
mainfrom
tlwu/gqa_batched_idle_skip_padding

Tianlei Wu (tianleiwu) commented Nov 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Ti-Tai Wang (titaiwangms) commented Nov 4, 2026

Uh oh!

Tianlei Wu (tianleiwu) commented Nov 4, 2026

Uh oh!

Ti-Tai Wang (titaiwangms) left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Chunk	Valid queries	Absolute totals
1	`[512, 512]`	`[512, 512]`
2	`[1, 512]`	`[513, 1024]`
3	`[0, 1]`	`[513, 1025]`

Conversation

Tianlei Wu (tianleiwu) commented Nov 3, 2026

Description

Summary of Changes

Length semantics and correctness

CPU

CUDA

Regression coverage and documentation

Testing

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Ti-Tai Wang (titaiwangms) commented Nov 4, 2026

Full review: changes recommended

Findings ledger

F1 — Skip CUDA tests when PyTorch has no CUDA support

Non-blocking open questions

Coverage, dissent, and residual risk

Uh oh!

Tianlei Wu (tianleiwu) commented Nov 4, 2026

Uh oh!

Ti-Tai Wang (titaiwangms) left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants