Repository navigation
Add BF16 support to CPU and CUDA Pad kernels - #34062
Merged
Merged
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add bfloat16 kernel for Pad node
Add BF16 support to CPU and CUDA Pad kernels
Nov 10, 2026
Tianlei Wu (tianleiwu)
marked this pull request as ready for review
November 10, 2026 04:43
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The updated BF16 registrations are missing from generated kernel documentation, causing validation to fail, and the CUDA regression test can falsely pass through CPU fallback.
2 open issues
What changed in this PR
Adds BF16 support to ONNX Runtime’s CPU and CUDA Pad kernels, addressing the session-initialization failure reported in #33874 for BF16 graphs that pad rotary caches.
Changes:
- Enables CPU BF16 Pad for opsets 21–25 with 16-bit constant-value handling.
- Adds CUDA BF16 registrations and kernel instantiations, including the optimized rank-2 path.
- Adds CPU- and CUDA-targeted tests for 2D bottom padding across opsets 21–25.
| File | Description |
|---|---|
onnxruntime/test/providers/cpu/tensor/pad_test.cc |
Adds BF16 padding regression tests for CPU and CUDA. |
onnxruntime/core/providers/cuda/tensor/pad.cc |
Instantiates BF16 Pad computation. |
onnxruntime/core/providers/cuda/tensor/pad_impl.cu |
Instantiates BF16 device kernels, including optimized padding. |
onnxruntime/core/providers/cuda/cuda_execution_provider.cc |
Registers BF16 Pad kernels with the CUDA provider. |
onnxruntime/core/providers/cpu/tensor/pad.cc |
Enables BF16 types and 16-bit value and computation dispatch. |
Review effort: Balanced
Add tensor(bfloat16) to the CPU Pad rows for opsets 21+ and to all CUDA Pad rows, matching the kernels registered by this PR. Generated with tools/python/gen_opkernel_doc.py (CPU and CUDA providers) on a local CUDA build and merged into the existing file; the DML and WebGPU sections are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RunCudaOnlyOnnxOpsetPadTest registered only the CUDA EP, but the CPU EP is always added implicitly, so the test could pass through CPU fallback when the CUDA kernel was missing. That became possible for BF16 now that the CPU EP also implements it. Set kOrtSessionOptionsDisableCPUEPFallback so a missing CUDA kernel fails session initialization instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 task done
This was referenced Nov 13, 2026
Franklin Moormann (ooples)
pushed a commit
to ooples/AiDotNet
that referenced
this pull request
Nov 13, 2026
Updated [Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.AspNetCore.Mvc.Testing's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Data.Sqlite's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime's releases](https://github.com/Microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime.Managed's releases](https://github.com/microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 18.11.0 to 18.12.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.NET.Test.Sdk's releases](https://github.com/microsoft/vstest/releases)._ ## 18.12.0 ## Security fix This release includes the fix for CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every version from 18.11.1 on is fixed, including 18.12.0. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Fix TestCaseStart/Stop events dropped for data-driven tests with shared TestCase.Id by @nohwnd in microsoft/vstest#16033 * Stop the discovery manager factory from shadowing discoveryCriteria by @nohwnd in microsoft/vstest#16391 * Gate the HTTP link checker on a fingerprint of the broken links by @nohwnd in microsoft/vstest#16413 * Quote boolean conditions in shipped Microsoft.TestPlatform.targets by @nohwnd in microsoft/vstest#16404 * Obsolete API hygiene: the compatibility-safe changes by @azat-msft in microsoft/vstest#16420 * Add xxHash128 test ids, available but not default by @azat-msft in microsoft/vstest#16378 * Isolate attachment manager test temporary files by @mmitche in microsoft/vstest#16463 * Test ids report logger by @azat-msft in microsoft/vstest#16443 * Isolate code coverage files in published output by @azat-msft in microsoft/vstest#16483 * Preserve OutputType for Android test projects by @jonathanpeppers in microsoft/vstest#16496 * Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft in microsoft/vstest#16462 * Mark the xxHash128 test id API as experimental by @azat-msft in microsoft/vstest#16465 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16633 * Fix localized resource paths in CLI source-build/non-Windows packages by @nohwnd in microsoft/vstest#16644 **Full Changelog**: microsoft/vstest@v18.11.0...v18.12.0 ## 18.11.1 ## Security fix This release fixes CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0. Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Security fix for CVE-2026-105544, see microsoft/vstest#16654 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16632 * Fix localized resource paths in CLI source-build packages by @nohwnd in microsoft/vstest#16643 **Full Changelog**: microsoft/vstest@v18.11.0...v18.11.1 Commits viewable in [compare view](microsoft/vstest@v18.11.0...v18.12.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Nov 14, 2026
Franklin Moormann (ooples)
pushed a commit
to ooples/AiDotNet
that referenced
this pull request
Nov 14, 2026
Updated [Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.AspNetCore.Mvc.Testing's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Data.Sqlite's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime's releases](https://github.com/Microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime.Managed's releases](https://github.com/microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 18.11.0 to 18.12.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.NET.Test.Sdk's releases](https://github.com/microsoft/vstest/releases)._ ## 18.12.0 ## Security fix This release includes the fix for CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every version from 18.11.1 on is fixed, including 18.12.0. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Fix TestCaseStart/Stop events dropped for data-driven tests with shared TestCase.Id by @nohwnd in microsoft/vstest#16033 * Stop the discovery manager factory from shadowing discoveryCriteria by @nohwnd in microsoft/vstest#16391 * Gate the HTTP link checker on a fingerprint of the broken links by @nohwnd in microsoft/vstest#16413 * Quote boolean conditions in shipped Microsoft.TestPlatform.targets by @nohwnd in microsoft/vstest#16404 * Obsolete API hygiene: the compatibility-safe changes by @azat-msft in microsoft/vstest#16420 * Add xxHash128 test ids, available but not default by @azat-msft in microsoft/vstest#16378 * Isolate attachment manager test temporary files by @mmitche in microsoft/vstest#16463 * Test ids report logger by @azat-msft in microsoft/vstest#16443 * Isolate code coverage files in published output by @azat-msft in microsoft/vstest#16483 * Preserve OutputType for Android test projects by @jonathanpeppers in microsoft/vstest#16496 * Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft in microsoft/vstest#16462 * Mark the xxHash128 test id API as experimental by @azat-msft in microsoft/vstest#16465 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16633 * Fix localized resource paths in CLI source-build/non-Windows packages by @nohwnd in microsoft/vstest#16644 **Full Changelog**: microsoft/vstest@v18.11.0...v18.12.0 ## 18.11.1 ## Security fix This release fixes CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0. Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Security fix for CVE-2026-105544, see microsoft/vstest#16654 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16632 * Fix localized resource paths in CLI source-build packages by @nohwnd in microsoft/vstest#16643 **Full Changelog**: microsoft/vstest@v18.11.0...v18.11.1 Commits viewable in [compare view](microsoft/vstest@v18.11.0...v18.12.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
BF16 Pad currently fails session initialization with CPU and CUDA providers, and CPU-only sessions report no implementation. Supporting it unblocks BF16 graphs that pad rotary caches without requiring a Concat workaround.