You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[CUDA] Accelerate external weight loading with pipelined pinned staging and plugin EP support - #33839
This PR accelerates external initializer loading to GPU memory for both the built-in CUDA Execution Provider and the CUDA Plugin Execution Provider without introducing external storage dependencies (libcufile / GPUDirect Storage or DirectStorage):
Replaces the per-chunk global thread-pool barrier and per-tensor cudaStreamSynchronize barrier with a shared lock-free work queue (ExecuteExternalDataLoadTasks in onnxruntime/core/framework/external_data_loader_thread_pool.h).
Each reader worker owns a 4 KiB-aligned slice in both double-buffered pinned staging buffers and a pair of non-blocking CUDA streams, pulling chunks (2 MiB default) via atomic index and overlapping multi-threaded file reads (pread on POSIX / OVERLAPPEDReadFile on Windows) with cudaMemcpyAsync across tensor boundaries.
CUDA Plugin EP (PluginExecutionProvider) Support:
Implements PluginExecutionProvider::GetExternalDataLoader() using host_accessible_memory_info (Reserve/Free for pinned memory), lightweight copy-only OrtSyncStreamImpl streams (copy_only=1, skipping unnecessary cuBLAS/cuDNN handle creation), and OrtDataTransferImpl (CopyTensorAsync).
Ensures CudaDataTransfer::CopyTensorsImpl and CudaSyncStream::FlushImpl select and restore the target CUDA device when invoked from reader worker threads.
Cross-Tensor Batch Lifecycle, Offset Sorting, and File-Handle Caching:
Extends IExternalDataLoader and ExternalDataLoaderManager with BeginLoad(), FinalizeLoad(), and EndLoad() so external tensors in SaveInitializedTensors are pipelined across tensor boundaries and synchronized once in FinalizeLoad().
Sorts initializers in SaveInitializedTensors by (external_location, external_offset, ort_value_index) for sequential file access instead of hash-bucket order, and caches validated RandomAccessFile handles per SaveInitializedTensors call.
Conservative Defaults, Model-Size Gating, Session Options, and Minimal Build Exclusion:
Computes the default reader thread count from physical CPU cores (clamp(physical_cores / 2, 2, 8)).
Uses a conservative 32 MiB default total pinned staging allocation (two 16 MiB buffers, 4x smaller than the previous 128 MiB default).
Defers pinned buffer, stream, and thread-pool allocation until FinalizeLoad() and gates it on a 16 MiB minimum total external initializer size on the device (session.external_data_loader.min_size_bytes), so small models incur zero pinned-memory or stream-creation overhead.
Adds session configuration keys (session.external_data_loader.threads, session.external_data_loader.pinned_memory_bytes, session.external_data_loader.chunk_bytes, session.external_data_loader.min_size_bytes) alongside matching CUDA EP provider options (external_data_loader_reading_threads, external_data_loader_pinned_memory_bytes, external_data_loader_chunk_bytes).
Excludes the feature from minimal builds (!defined(ORT_MINIMAL_BUILD) and !defined(USE_CUDA_MINIMAL)).
Add C++ and Python unit tests for session options, provider options, min-size gating, and invalid-option rejection
Motivation and Context
Loading large ONNX models with external weights (e.g., qwen_flash_next_int4_int8_rc5_contrib_dflash2/model.onnx, 68.46 GiB across 899 external tensors in 7 .data files) into GPU memory during InferenceSession creation had several bottlenecks:
PluginExecutionProvider (used when building the CUDA EP as a plugin) did not implement GetExternalDataLoader(), falling back to per-tensor mmap + single-threaded pageable cudaMemcpy.
In the existing pinned-buffer loader (Load CUDA external data through pinned buffers #32437 / Add CUDA weight loading with GPUDirect Storage and Microsoft DirectStorage #32713 fallback when nvidia-fs is unavailable), a global thread-pool barrier per 64 MiB chunk and a cudaStreamSynchronize barrier at the end of every LoadTensor() call prevented CPU/GPU overlap across tensors <= 64 MiB (705 of 899 tensors), forced all 613 tensors < 16 MiB onto a single thread, iterated over initializers in arbitrary hash-bucket order, and reopened/revalidated .data files for every tensor.
This PR (same 4 threads, same 128 MiB pinned memory)
4
128 MiB
3.657 s
18.72 GiB/s
3.96x
This PR (Default: 8 threads from CPU cores, conservative 32 MiB pinned)
8
32 MiB
2.191 s
31.25 GiB/s
6.61x
This PR (Opt-in: 16 threads, 64 MiB pinned)
16
64 MiB
1.967 s
34.81 GiB/s
7.36x
Even at the exact same 4-thread count and 4x smaller pinned memory (32 MiB vs 128 MiB), the lock-free cross-tensor pipeline + offset sorting + file-handle caching reduces session creation time from 5.110 s to 3.730 s (1.37x faster). With the CPU-core-derived default (8 threads) and conservative 32 MiB pinned memory, session creation completes in 2.191 s (31.25 GiB/s).
…ng and plugin EP support
- Replace per-chunk thread-pool barriers and per-tensor stream synchronizations with a lock-free multi-worker double-buffered pinned staging and async H2D copy pipeline across external initializers.
- Add external data loader support to PluginExecutionProvider (including CUDA plugin EP) via host-accessible pinned memory, lightweight copy-only OrtSyncStreamImpl streams, and async OrtDataTransferImpl copies.
- Sort external initializers by (location, offset) and cache validated RandomAccessFile handles across SaveInitializedTensors.
- Compute default reader thread count from physical CPU cores (clamped to [2, 8]), use a conservative 32 MiB total pinned staging memory default (two 16 MiB buffers), and gate pinned staging allocation on a 16 MiB minimum total external weight threshold.
- Add session configuration keys (session.external_data_loader.threads, session.external_data_loader.pinned_memory_bytes, session.external_data_loader.chunk_bytes, session.external_data_loader.min_size_bytes) and matching CUDA EP provider options; exclude feature from minimal builds.
Co-authored-by: Xavier Dupré <xadupre@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unresolved critical issues allow GPU copies to outlive their buffers and staging memory to be reused prematurely, while minimal and external-initializer-disabled build configurations fail to compile.
This PR extends ONNX Runtime’s pinned-buffer external-weight loader to pipeline reads and GPU copies across tensors, including support for the CUDA plugin execution provider.
- Drain async copy streams when a read or callback throws in both serial and parallel paths, and add tests.
- Guard the CUDA thread-pool alias and plugin loader option validation for minimal builds.
- Drain pending CUDA default-stream copies before switching devices in plugin CopyTensors.
- Include test asserts header unconditionally.
- Bound cached external data file handles by flushing loaders before releasing them.
- Preserve explicit provider option presence through OrtCUDAProviderOptionsV2.
- Assert MatMul nodes are assigned to the plugin EP in the external data loader test.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Critical loader defects can select an invalid CUDA copy direction, claim unsupported plugin transfers, and exhaust host memory by materializing every chunk descriptor.
The built-in CUDA provider compiles this helper with SHARED_PROVIDER, so its default uses hardware_concurrency() / 4 rather than the documented physical-core count divided by two. On a 16-core host without simultaneous multithreading, it selects four readers while the plugin selects eight, reducing default loading parallelism. Resolve the physical-core count through the provider bridge or pass the core-computed default into the CUDA loader.
These loops validate bare external-loader keys and ep.cuda.* aliases, but the loader only consumes the actual EP-prefixed provider options and session.external_data_loader.* entries. Setting a valid bare key such as external_data_loader_reading_threads=8 therefore has no effect, even though invalid values are rejected. Forward the accepted aliases to the loader, or remove aliases that are not supported.
- Lower the pinned staging memory limit to 1 GiB and rename it to kMaxExternalDataLoaderPinnedMemoryBytes.
- Generate parallel read chunks lazily instead of materializing every chunk descriptor.
- Treat host-accessible (pinned) GPU-typed memory as host memory in the CUDA plugin data transfer.
- Enable the plugin external data loader only when the plugin data transfer can copy pinned and pageable host memory to the device.
- Gate pinned staging on the total external bytes (cumulative across flushes and reported up front by the framework) instead of each partial batch.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The direct Tensor call breaks Windows CUDA test linking, while registering the CUDA-style loader for arbitrary GPU plugins risks invalid device-data access.
This CUDA-style pipeline is registered for any GPU plugin, but CanCopy does not guarantee byte-addressable device data or compatible allocator and stream callbacks. Plugins using opaque buffer handles, allocator-info pointer identity, or session-owned streams can receive invalid slices or always fall back to pageable loading. Restrict registration to CUDA until these additional capabilities are explicitly negotiated.
Test cache flushes with deferred loads and more than 64 files
There is no session-level regression test for the cache's 64-file flush. Pending loads retain raw file pointers, so clearing the cache before every loader finishes would make later reads use freed handles. Add a session test with more than 64 external files and a deferred tracking loader that verifies file/offset ordering, mid-load finalization, and successful loading.
Preserve flat CUDA loader aliases when extracting provider options
The factory validates the flat external_data_loader_* session-config aliases, but provider_options_ only captures EP-prefixed keys, so a flat external_data_loader_reading_threads=0 has no effect. Preserve the flat aliases when extracting CUDA provider options, with EP-prefixed values retaining precedence. The same issue affects the pinned-memory and chunk aliases.
Resolve automatic reader count from host CPU in session initialization
With no explicit thread option, shared CUDA uses logical CPU threads / 4 rather than physical cores / 2, and batches below one staging buffer still select the fixed count of four. This can produce four readers instead of eight on a 16-core host without SMT. Resolve the automatic count in InferenceSession::Initialize() and pass it through Configure() so both reader counts receive the host-derived value.
Report the actual or automatic reader count in benchmark JSON
When --reading-threads is omitted, the benchmark now uses the runtime's automatic reader selection, but its JSON still reports reading_threads: 4 at line 123. This mislabels runs that select another count; report the actual count if available, or mark the setting as automatic rather than inventing a numeric value.
- Add an open-file LoadCudaExternalDataForTest bridge overload and route batched test loads through it to avoid core/shared-provider Tensor symbol mismatches on MSVC.
- Restrict plugin external data loader registration to CUDA plugin EPs on NVIDIA GPU devices and preserve ep.cudaexecutionprovider.* and flat session-config option aliases.
- Add mock-plugin unit tests in ep_plugin_provider_test.cc covering missing callbacks, null/incomplete data transfers, rejected host-to-device CanCopy pairs, option aliases, and disablement.
- Add a 66-file deferred-read session test in external_data_loader_test.cc verifying external file cache flushes drain queued loads before releasing file handles.
- Resolve automatic reader count from host CPU in InferenceSession::Initialize() and pass it through Configure() so both single-tensor and batch reader counts receive the host-derived value.
- Report "auto" for reading_threads in benchmark_cuda_model_loading.py when --reading-threads is omitted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a unique temp-file prefix per file in the 66-file test since
_wmktemp_s supports only 26 names per template, and skip the
threads=0 fallback test on WASM where Module.MountedFiles is unavailable.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updated
[Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet)
from 10.0.14 to 10.0.15.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.AspNetCore.Mvc.Testing's
releases](https://github.com/dotnet/dotnet/releases)._
No release notes found for this version range.
Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>
Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from
10.0.14 to 10.0.15.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.Data.Sqlite's
releases](https://github.com/dotnet/dotnet/releases)._
No release notes found for this version range.
Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>
Updated
[Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime)
from 1.31.1 to 1.32.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.ML.OnnxRuntime's
releases](https://github.com/Microsoft/onnxruntime/releases)._
## 1.32.0
ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support,
introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE
kernels, expands device-memory, weightless-compilation, and Python
model-package APIs, dramatically accelerates session initialization on
large graphs, and strengthens runtime and optimizer reliability. These
notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel
updates are summarized here; detailed provider notes are covered by
their separate plugin EP releases.
## Highlights
- Reintegrated **ONNX 1.23.2** with **opset 28** operator support
(`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions)
and updated **DLPack to v1.3** across all implemented ORT numeric and
boolean tensor types
([#33134](microsoft/onnxruntime#33134),
[#32362](microsoft/onnxruntime#32362),
[#33848](microsoft/onnxruntime#33848)).
- Added a session-scoped **CPU/GPU hybrid MoE offload** planner and
execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and
block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2
CPU `QMoE` kernels and memory-mapped expert storage
([#33450](microsoft/onnxruntime#33450),
[#33451](microsoft/onnxruntime#33451),
[#33452](microsoft/onnxruntime#33452),
[#33973](microsoft/onnxruntime#33973)).
- Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit
multi-mode weightless compilation
(`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode`), Python bindings for the Model Package
APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers`
([#33447](microsoft/onnxruntime#33447),
[#32899](microsoft/onnxruntime#32899),
[#33640](microsoft/onnxruntime#33640),
[#33877](microsoft/onnxruntime#33877)).
- Eliminated superlinear bottlenecks in allocation planning, constant
sharing, graph I/O mapping, and topological sorting—reducing session
initialization on 100k-node constant-heavy graphs from minutes to
seconds—and added pipelined pinned-memory staging for external GPU
weight loading
([#33407](microsoft/onnxruntime#33407),
[#33939](microsoft/onnxruntime#33939),
[#34009](microsoft/onnxruntime#34009),
[#33839](microsoft/onnxruntime#33839)).
- Expanded shared generative AI and quantized contrib operators,
including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension
decay), right-padded chunked prefill in `GroupQueryAttention`, native
GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`,
`MoEPacked`), and sparse-attention index sharing
([#33817](microsoft/onnxruntime#33817),
[#33503](microsoft/onnxruntime#33503),
[#33953](microsoft/onnxruntime#33953),
[#33224](microsoft/onnxruntime#33224)).
## Announcements & Compatibility
- **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched
with ONNX Runtime 1.32.0
([#33214](microsoft/onnxruntime#33214)). For
**contrib-op compatibility**, core and the plugin must use the same
contributed-operator schemas; building both from the **same ONNX Runtime
revision** is recommended. Schema mismatches can cause incorrect
execution or crashes and may not be detected during plugin registration.
- **ACL EP removed:** Following its deprecation in 1.31.0, the Arm
Compute Library (`ACLExecutionProvider`) execution provider, its build
options, and its standalone C registration API have been removed.
Python's `get_all_providers()` no longer lists `ACLExecutionProvider`;
applications should remove it from explicit provider lists
([#33851](microsoft/onnxruntime#33851)).
- **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:**
In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have
been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP
registered via `OrtApi::RegisterExecutionProviderLibrary`.
`NvTensorRtRtxProviderOptions` is retained as a deprecated declaration
for source compatibility, and legacy `AppendExecutionProvider` calls now
report a deprecation error directing callers to the plugin EP
([#32276](microsoft/onnxruntime#32276)).
- **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**,
raising supported opset ceilings to **28** and restoring
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings
and `Cast` conversions
([#33134](microsoft/onnxruntime#33134)).
- **DLPack v1.3 boolean contract change:** DLPack has been upgraded to
**v1.3** with native `kDLBool` metadata and bidirectional support for
all implemented ORT numeric and boolean types, including all five
`float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2`
(`float6` remains unsupported in DLPack). The low-level `is_bool_tensor`
override was removed, and `uint8` DLPack metadata now always imports as
`uint8`
([#33848](microsoft/onnxruntime#33848)).
- **Weightless compilation API migration:**
`OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the
`ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session
options are deprecated in favor of
`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`,
`"2"` = `ALL`). EPs can now advertise multi-mode capability via
`OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and
`weightless_supported_modes`
([#32899](microsoft/onnxruntime#32899)).
- **Training CI retirement:** Training build and test lanes were retired
from active GitHub Actions CI workflows while retaining inference build
coverage
([#33836](microsoft/onnxruntime#33836)).
- This is the last release for [CocoaPods](https://cocoapods.org/), and
we will switch to [Swift Package
Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/).
- We will use combo onnxruntime-gpu python package to reduce storage
size from next release. You can test it
[here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview)
## New Features
### Core APIs & Runtime
- Added `OrtApi::EpDevice_GetMemoryCapacity` (C++
`EpDevice::GetMemoryCapacity()`) and
`OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total
memory and estimated currently available memory (`DEFAULT` or
`HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support
([#33447](microsoft/onnxruntime#33447)).
- Expanded the app–EP weightless contract so EPs advertise all supported
weightless modes (`weightless_supported_modes`), applications request an
explicit mode (`ModelCompilationOptions_SetWeightlessMode` /
`ep.enable_weightless_mode`), and compiled EPContext models record
`weightless_mode` and `external_dependencies` (`source_model`,
`external_weights`, `ep_context_binary`) in model metadata
([#32899](microsoft/onnxruntime#32899)).
- Extended `session.use_external_initializer_file_buffers_directly="1"`
to `AddExternalInitializers` for zero-copy borrowing of naturally
aligned CPU buffers larger than 127 bytes on little-endian hosts, and
documented the read-only guarantee for external initializers added via
`OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true`
([#33877](microsoft/onnxruntime#33877),
[#33717](microsoft/onnxruntime#33717)).
- Added lock-free multi-worker pinned-memory staging
(`session.external_data_loader.*`) to accelerate external initializer
loading into GPU memory across built-in and plugin CUDA EPs
([#33839](microsoft/onnxruntime#33839)).
- Added a session-scoped MoE/QMoE CPU/GPU offload planner and
memory-budget ledger (`session.moe.offload.*`), generic
`session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`)
configuration plumbing, Level-1 workspace/prepack memory accounting
fallback for nodes absent from memory-profile CSVs, and internal
capability-planning protocol types
([#33451](microsoft/onnxruntime#33451),
[#33042](microsoft/onnxruntime#33042),
[#33311](microsoft/onnxruntime#33311),
[#33841](microsoft/onnxruntime#33841)).
- Enabled cross-thread reuse of idle device streams when graph capture
is disabled and providers register device-selection callbacks, and
enabled compile-session profiling (`<profile_file>_compile`) in
`onnxruntime_perf_test` when `-p` is used with `--compile_ep_context`
([#33053](microsoft/onnxruntime#33053),
[#33694](microsoft/onnxruntime#33694)).
- Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable
without `onnxruntime_MINIMAL_BUILD` and added the
`onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream
support in non-minimal builds
([#33120](microsoft/onnxruntime#33120),
[#33482](microsoft/onnxruntime#33482)).
### ONNX Opset 28 & Shared Contrib Operators
These updates affect core ONNX and contributed operator schemas and
cross-provider behavior, not only the separately released GPU plugins.
- Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and
`CRD`) on CPU and CUDA with version-split registrations across DML, JS,
WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and
DML; opset-28 `BFloat16` support for `Compress`, `OneHot`,
`ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus
`Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16`
`Pad` (opsets 21–25) on CPU and CUDA
([#32362](microsoft/onnxruntime#32362),
[#32369](microsoft/onnxruntime#32369),
[#32378](microsoft/onnxruntime#32378),
[#34062](microsoft/onnxruntime#34062)).
- Extended `GatedDeltaNet` with Kimi Delta Attention (KDA)
per-key-dimension decay across CPU, CUDA, and WebGPU, and extended
`GatedDeltaNet` and `CausalConvWithState` state/gate handling for
multi-token speculative decoding
([#33817](microsoft/onnxruntime#33817),
[#33471](microsoft/onnxruntime#33471)).
- Added an optional `query_seqlens` input to `GroupQueryAttention` to
skip right-padded prefill tokens and idle batch entries
(`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` /
`k_norm_epsilon` attributes to preserve distinct Q and K normalization
epsilons during GQA pre-norm fusion
([#33503](microsoft/onnxruntime#33503),
[#33505](microsoft/onnxruntime#33505),
[#33832](microsoft/onnxruntime#33832),
[#33363](microsoft/onnxruntime#33363)).
- Defined the INT3 `QMoE` v1 packed-code contract, added opt-in
post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU`
(`activation_type="geglu"`) support to MoE/QMoE schemas, and added
`GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt
`global_scale`), and `MoEPacked` (CUDA) to consume native
GGUF/block-packed quantized weights directly
([#33394](microsoft/onnxruntime#33394),
[#33954](microsoft/onnxruntime#33954),
[#33633](microsoft/onnxruntime#33633),
[#33953](microsoft/onnxruntime#33953),
[#33962](microsoft/onnxruntime#33962)).
- Added `PackedSparseAttentionIndexerMerge` for multi-token prediction
index sharing and optional `state_status` / mid-state outputs on
`PackedSparseAttentionIndexer` across CUDA and WebGPU; extended
`GatedAdd` with an optional projected `gate_weight` across CPU, CUDA,
and WebGPU; and supported empty unpacked weights in CPU and WebGPU
`MatMulNBits`
([#33224](microsoft/onnxruntime#33224),
[#33383](microsoft/onnxruntime#33383),
[#33472](microsoft/onnxruntime#33472),
[#33470](microsoft/onnxruntime#33470),
[#33559](microsoft/onnxruntime#33559)).
- Registered core contrib schemas for provider-targeted LLM and
speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`,
`HyperConnectionLowRankMix`, `RouterGateUpProj`,
`SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`,
`ConvStateReplay`, and `MatMulNBitsQLoRA`
([#32126](microsoft/onnxruntime#32126),
[#33469](microsoft/onnxruntime#33469),
[#33384](microsoft/onnxruntime#33384),
[#33650](microsoft/onnxruntime#33650),
[#33474](microsoft/onnxruntime#33474),
[#33213](microsoft/onnxruntime#33213),
[#33493](microsoft/onnxruntime#33493),
[#33370](microsoft/onnxruntime#33370)).
### Plugin Execution Provider Infrastructure
- Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP
boundary by converting `OrtMemoryInfo` instances exclusively through the
C API ([#33571](microsoft/onnxruntime#33571)).
- Zero-initialized `OrtEpFactory` structs in provider-bridge EP
factories so unimplemented optional callbacks are null, and prevented
double DLL initialization and handle leaks when probing and registering
plugin EP libraries
([#33117](microsoft/onnxruntime#33117),
[#33701](microsoft/onnxruntime#33701)).
- Hardened portable model-package EP recreation, custom-domain
preservation, UTF-8 path handling, and `shared_assets` path confinement
and symlink validation
([#32790](microsoft/onnxruntime#32790),
[#33826](microsoft/onnxruntime#33826)).
... (truncated)
Commits viewable in [compare
view](microsoft/onnxruntime@v1.31.1...v1.32.0).
</details>
Updated
[Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime)
from 1.31.1 to 1.32.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.ML.OnnxRuntime.Managed's
releases](https://github.com/microsoft/onnxruntime/releases)._
## 1.32.0
ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support,
introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE
kernels, expands device-memory, weightless-compilation, and Python
model-package APIs, dramatically accelerates session initialization on
large graphs, and strengthens runtime and optimizer reliability. These
notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel
updates are summarized here; detailed provider notes are covered by
their separate plugin EP releases.
## Highlights
- Reintegrated **ONNX 1.23.2** with **opset 28** operator support
(`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions)
and updated **DLPack to v1.3** across all implemented ORT numeric and
boolean tensor types
([#33134](microsoft/onnxruntime#33134),
[#32362](microsoft/onnxruntime#32362),
[#33848](microsoft/onnxruntime#33848)).
- Added a session-scoped **CPU/GPU hybrid MoE offload** planner and
execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and
block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2
CPU `QMoE` kernels and memory-mapped expert storage
([#33450](microsoft/onnxruntime#33450),
[#33451](microsoft/onnxruntime#33451),
[#33452](microsoft/onnxruntime#33452),
[#33973](microsoft/onnxruntime#33973)).
- Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit
multi-mode weightless compilation
(`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode`), Python bindings for the Model Package
APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers`
([#33447](microsoft/onnxruntime#33447),
[#32899](microsoft/onnxruntime#32899),
[#33640](microsoft/onnxruntime#33640),
[#33877](microsoft/onnxruntime#33877)).
- Eliminated superlinear bottlenecks in allocation planning, constant
sharing, graph I/O mapping, and topological sorting—reducing session
initialization on 100k-node constant-heavy graphs from minutes to
seconds—and added pipelined pinned-memory staging for external GPU
weight loading
([#33407](microsoft/onnxruntime#33407),
[#33939](microsoft/onnxruntime#33939),
[#34009](microsoft/onnxruntime#34009),
[#33839](microsoft/onnxruntime#33839)).
- Expanded shared generative AI and quantized contrib operators,
including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension
decay), right-padded chunked prefill in `GroupQueryAttention`, native
GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`,
`MoEPacked`), and sparse-attention index sharing
([#33817](microsoft/onnxruntime#33817),
[#33503](microsoft/onnxruntime#33503),
[#33953](microsoft/onnxruntime#33953),
[#33224](microsoft/onnxruntime#33224)).
## Announcements & Compatibility
- **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched
with ONNX Runtime 1.32.0
([#33214](microsoft/onnxruntime#33214)). For
**contrib-op compatibility**, core and the plugin must use the same
contributed-operator schemas; building both from the **same ONNX Runtime
revision** is recommended. Schema mismatches can cause incorrect
execution or crashes and may not be detected during plugin registration.
- **ACL EP removed:** Following its deprecation in 1.31.0, the Arm
Compute Library (`ACLExecutionProvider`) execution provider, its build
options, and its standalone C registration API have been removed.
Python's `get_all_providers()` no longer lists `ACLExecutionProvider`;
applications should remove it from explicit provider lists
([#33851](microsoft/onnxruntime#33851)).
- **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:**
In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have
been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP
registered via `OrtApi::RegisterExecutionProviderLibrary`.
`NvTensorRtRtxProviderOptions` is retained as a deprecated declaration
for source compatibility, and legacy `AppendExecutionProvider` calls now
report a deprecation error directing callers to the plugin EP
([#32276](microsoft/onnxruntime#32276)).
- **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**,
raising supported opset ceilings to **28** and restoring
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings
and `Cast` conversions
([#33134](microsoft/onnxruntime#33134)).
- **DLPack v1.3 boolean contract change:** DLPack has been upgraded to
**v1.3** with native `kDLBool` metadata and bidirectional support for
all implemented ORT numeric and boolean types, including all five
`float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2`
(`float6` remains unsupported in DLPack). The low-level `is_bool_tensor`
override was removed, and `uint8` DLPack metadata now always imports as
`uint8`
([#33848](microsoft/onnxruntime#33848)).
- **Weightless compilation API migration:**
`OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the
`ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session
options are deprecated in favor of
`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`,
`"2"` = `ALL`). EPs can now advertise multi-mode capability via
`OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and
`weightless_supported_modes`
([#32899](microsoft/onnxruntime#32899)).
- **Training CI retirement:** Training build and test lanes were retired
from active GitHub Actions CI workflows while retaining inference build
coverage
([#33836](microsoft/onnxruntime#33836)).
- This is the last release for [CocoaPods](https://cocoapods.org/), and
we will switch to [Swift Package
Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/).
- We will use combo onnxruntime-gpu python package to reduce storage
size from next release. You can test it
[here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview)
## New Features
### Core APIs & Runtime
- Added `OrtApi::EpDevice_GetMemoryCapacity` (C++
`EpDevice::GetMemoryCapacity()`) and
`OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total
memory and estimated currently available memory (`DEFAULT` or
`HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support
([#33447](microsoft/onnxruntime#33447)).
- Expanded the app–EP weightless contract so EPs advertise all supported
weightless modes (`weightless_supported_modes`), applications request an
explicit mode (`ModelCompilationOptions_SetWeightlessMode` /
`ep.enable_weightless_mode`), and compiled EPContext models record
`weightless_mode` and `external_dependencies` (`source_model`,
`external_weights`, `ep_context_binary`) in model metadata
([#32899](microsoft/onnxruntime#32899)).
- Extended `session.use_external_initializer_file_buffers_directly="1"`
to `AddExternalInitializers` for zero-copy borrowing of naturally
aligned CPU buffers larger than 127 bytes on little-endian hosts, and
documented the read-only guarantee for external initializers added via
`OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true`
([#33877](microsoft/onnxruntime#33877),
[#33717](microsoft/onnxruntime#33717)).
- Added lock-free multi-worker pinned-memory staging
(`session.external_data_loader.*`) to accelerate external initializer
loading into GPU memory across built-in and plugin CUDA EPs
([#33839](microsoft/onnxruntime#33839)).
- Added a session-scoped MoE/QMoE CPU/GPU offload planner and
memory-budget ledger (`session.moe.offload.*`), generic
`session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`)
configuration plumbing, Level-1 workspace/prepack memory accounting
fallback for nodes absent from memory-profile CSVs, and internal
capability-planning protocol types
([#33451](microsoft/onnxruntime#33451),
[#33042](microsoft/onnxruntime#33042),
[#33311](microsoft/onnxruntime#33311),
[#33841](microsoft/onnxruntime#33841)).
- Enabled cross-thread reuse of idle device streams when graph capture
is disabled and providers register device-selection callbacks, and
enabled compile-session profiling (`<profile_file>_compile`) in
`onnxruntime_perf_test` when `-p` is used with `--compile_ep_context`
([#33053](microsoft/onnxruntime#33053),
[#33694](microsoft/onnxruntime#33694)).
- Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable
without `onnxruntime_MINIMAL_BUILD` and added the
`onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream
support in non-minimal builds
([#33120](microsoft/onnxruntime#33120),
[#33482](microsoft/onnxruntime#33482)).
### ONNX Opset 28 & Shared Contrib Operators
These updates affect core ONNX and contributed operator schemas and
cross-provider behavior, not only the separately released GPU plugins.
- Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and
`CRD`) on CPU and CUDA with version-split registrations across DML, JS,
WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and
DML; opset-28 `BFloat16` support for `Compress`, `OneHot`,
`ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus
`Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16`
`Pad` (opsets 21–25) on CPU and CUDA
([#32362](microsoft/onnxruntime#32362),
[#32369](microsoft/onnxruntime#32369),
[#32378](microsoft/onnxruntime#32378),
[#34062](microsoft/onnxruntime#34062)).
- Extended `GatedDeltaNet` with Kimi Delta Attention (KDA)
per-key-dimension decay across CPU, CUDA, and WebGPU, and extended
`GatedDeltaNet` and `CausalConvWithState` state/gate handling for
multi-token speculative decoding
([#33817](microsoft/onnxruntime#33817),
[#33471](microsoft/onnxruntime#33471)).
- Added an optional `query_seqlens` input to `GroupQueryAttention` to
skip right-padded prefill tokens and idle batch entries
(`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` /
`k_norm_epsilon` attributes to preserve distinct Q and K normalization
epsilons during GQA pre-norm fusion
([#33503](microsoft/onnxruntime#33503),
[#33505](microsoft/onnxruntime#33505),
[#33832](microsoft/onnxruntime#33832),
[#33363](microsoft/onnxruntime#33363)).
- Defined the INT3 `QMoE` v1 packed-code contract, added opt-in
post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU`
(`activation_type="geglu"`) support to MoE/QMoE schemas, and added
`GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt
`global_scale`), and `MoEPacked` (CUDA) to consume native
GGUF/block-packed quantized weights directly
([#33394](microsoft/onnxruntime#33394),
[#33954](microsoft/onnxruntime#33954),
[#33633](microsoft/onnxruntime#33633),
[#33953](microsoft/onnxruntime#33953),
[#33962](microsoft/onnxruntime#33962)).
- Added `PackedSparseAttentionIndexerMerge` for multi-token prediction
index sharing and optional `state_status` / mid-state outputs on
`PackedSparseAttentionIndexer` across CUDA and WebGPU; extended
`GatedAdd` with an optional projected `gate_weight` across CPU, CUDA,
and WebGPU; and supported empty unpacked weights in CPU and WebGPU
`MatMulNBits`
([#33224](microsoft/onnxruntime#33224),
[#33383](microsoft/onnxruntime#33383),
[#33472](microsoft/onnxruntime#33472),
[#33470](microsoft/onnxruntime#33470),
[#33559](microsoft/onnxruntime#33559)).
- Registered core contrib schemas for provider-targeted LLM and
speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`,
`HyperConnectionLowRankMix`, `RouterGateUpProj`,
`SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`,
`ConvStateReplay`, and `MatMulNBitsQLoRA`
([#32126](microsoft/onnxruntime#32126),
[#33469](microsoft/onnxruntime#33469),
[#33384](microsoft/onnxruntime#33384),
[#33650](microsoft/onnxruntime#33650),
[#33474](microsoft/onnxruntime#33474),
[#33213](microsoft/onnxruntime#33213),
[#33493](microsoft/onnxruntime#33493),
[#33370](microsoft/onnxruntime#33370)).
### Plugin Execution Provider Infrastructure
- Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP
boundary by converting `OrtMemoryInfo` instances exclusively through the
C API ([#33571](microsoft/onnxruntime#33571)).
- Zero-initialized `OrtEpFactory` structs in provider-bridge EP
factories so unimplemented optional callbacks are null, and prevented
double DLL initialization and handle leaks when probing and registering
plugin EP libraries
([#33117](microsoft/onnxruntime#33117),
[#33701](microsoft/onnxruntime#33701)).
- Hardened portable model-package EP recreation, custom-domain
preservation, UTF-8 path handling, and `shared_assets` path confinement
and symlink validation
([#32790](microsoft/onnxruntime#32790),
[#33826](microsoft/onnxruntime#33826)).
... (truncated)
Commits viewable in [compare
view](microsoft/onnxruntime@v1.31.1...v1.32.0).
</details>
Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest)
from 18.11.0 to 18.12.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.NET.Test.Sdk's
releases](https://github.com/microsoft/vstest/releases)._
## 18.12.0
## Security fix
This release includes the fix for CVE-2026-105544, an elevation of
privilege vulnerability in the VSTest artifact post-processor. The
post-processor could consume a forged `executionComplete.json` written
by another local user, and then run attacker-controlled output handling.
Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every
version from 18.11.1 on is fixed, including 18.12.0.
`Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the
.NET SDK. The recommended way to get this fix is to update your .NET SDK
to the latest patched version, see the advisory for the SDK versions.
The fix is also released in the `Microsoft.TestPlatform.CLI` package,
for completeness. That package is rarely used directly. It is not a
dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so
those packages do not bring it in.
Details: microsoft/vstest#16654 and
dotnet/announcements#517
Using custom Docker images or Nano Server? See
microsoft/vstest#16652
## What's Changed
* Fix TestCaseStart/Stop events dropped for data-driven tests with
shared TestCase.Id by @nohwnd in
microsoft/vstest#16033
* Stop the discovery manager factory from shadowing discoveryCriteria by
@nohwnd in microsoft/vstest#16391
* Gate the HTTP link checker on a fingerprint of the broken links by
@nohwnd in microsoft/vstest#16413
* Quote boolean conditions in shipped Microsoft.TestPlatform.targets by
@nohwnd in microsoft/vstest#16404
* Obsolete API hygiene: the compatibility-safe changes by @azat-msft in
microsoft/vstest#16420
* Add xxHash128 test ids, available but not default by @azat-msft in
microsoft/vstest#16378
* Isolate attachment manager test temporary files by @mmitche in
microsoft/vstest#16463
* Test ids report logger by @azat-msft in
microsoft/vstest#16443
* Isolate code coverage files in published output by @azat-msft in
microsoft/vstest#16483
* Preserve OutputType for Android test projects by @jonathanpeppers in
microsoft/vstest#16496
* Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft
in microsoft/vstest#16462
* Mark the xxHash128 test id API as experimental by @azat-msft in
microsoft/vstest#16465
* Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform
and Portable packages by @nohwnd in
microsoft/vstest#16633
* Fix localized resource paths in CLI source-build/non-Windows packages
by @nohwnd in microsoft/vstest#16644
**Full Changelog**:
microsoft/vstest@v18.11.0...v18.12.0
## 18.11.1
## Security fix
This release fixes CVE-2026-105544, an elevation of privilege
vulnerability in the VSTest artifact post-processor. The post-processor
could consume a forged `executionComplete.json` written by another local
user, and then run attacker-controlled output handling.
Affected: `Microsoft.TestPlatform.CLI` 18.11.0.
Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1.
`Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the
.NET SDK. The recommended way to get this fix is to update your .NET SDK
to the latest patched version, see the advisory for the SDK versions.
The fix is also released in the `Microsoft.TestPlatform.CLI` package,
for completeness. That package is rarely used directly. It is not a
dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so
those packages do not bring it in.
Details: microsoft/vstest#16654 and
dotnet/announcements#517
Using custom Docker images or Nano Server? See
microsoft/vstest#16652
## What's Changed
* Security fix for CVE-2026-105544, see
microsoft/vstest#16654
* Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform
and Portable packages by @nohwnd in
microsoft/vstest#16632
* Fix localized resource paths in CLI source-build packages by @nohwnd
in microsoft/vstest#16643
**Full Changelog**:
microsoft/vstest@v18.11.0...v18.11.1
Commits viewable in [compare
view](microsoft/vstest@v18.11.0...v18.12.0).
</details>
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updated
[Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet)
from 10.0.14 to 10.0.15.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.AspNetCore.Mvc.Testing's
releases](https://github.com/dotnet/dotnet/releases)._
No release notes found for this version range.
Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>
Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from
10.0.14 to 10.0.15.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.Data.Sqlite's
releases](https://github.com/dotnet/dotnet/releases)._
No release notes found for this version range.
Commits viewable in [compare
view](https://github.com/dotnet/dotnet/commits).
</details>
Updated
[Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime)
from 1.31.1 to 1.32.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.ML.OnnxRuntime's
releases](https://github.com/Microsoft/onnxruntime/releases)._
## 1.32.0
ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support,
introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE
kernels, expands device-memory, weightless-compilation, and Python
model-package APIs, dramatically accelerates session initialization on
large graphs, and strengthens runtime and optimizer reliability. These
notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel
updates are summarized here; detailed provider notes are covered by
their separate plugin EP releases.
## Highlights
- Reintegrated **ONNX 1.23.2** with **opset 28** operator support
(`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions)
and updated **DLPack to v1.3** across all implemented ORT numeric and
boolean tensor types
([#33134](microsoft/onnxruntime#33134),
[#32362](microsoft/onnxruntime#32362),
[#33848](microsoft/onnxruntime#33848)).
- Added a session-scoped **CPU/GPU hybrid MoE offload** planner and
execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and
block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2
CPU `QMoE` kernels and memory-mapped expert storage
([#33450](microsoft/onnxruntime#33450),
[#33451](microsoft/onnxruntime#33451),
[#33452](microsoft/onnxruntime#33452),
[#33973](microsoft/onnxruntime#33973)).
- Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit
multi-mode weightless compilation
(`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode`), Python bindings for the Model Package
APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers`
([#33447](microsoft/onnxruntime#33447),
[#32899](microsoft/onnxruntime#32899),
[#33640](microsoft/onnxruntime#33640),
[#33877](microsoft/onnxruntime#33877)).
- Eliminated superlinear bottlenecks in allocation planning, constant
sharing, graph I/O mapping, and topological sorting—reducing session
initialization on 100k-node constant-heavy graphs from minutes to
seconds—and added pipelined pinned-memory staging for external GPU
weight loading
([#33407](microsoft/onnxruntime#33407),
[#33939](microsoft/onnxruntime#33939),
[#34009](microsoft/onnxruntime#34009),
[#33839](microsoft/onnxruntime#33839)).
- Expanded shared generative AI and quantized contrib operators,
including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension
decay), right-padded chunked prefill in `GroupQueryAttention`, native
GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`,
`MoEPacked`), and sparse-attention index sharing
([#33817](microsoft/onnxruntime#33817),
[#33503](microsoft/onnxruntime#33503),
[#33953](microsoft/onnxruntime#33953),
[#33224](microsoft/onnxruntime#33224)).
## Announcements & Compatibility
- **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched
with ONNX Runtime 1.32.0
([#33214](microsoft/onnxruntime#33214)). For
**contrib-op compatibility**, core and the plugin must use the same
contributed-operator schemas; building both from the **same ONNX Runtime
revision** is recommended. Schema mismatches can cause incorrect
execution or crashes and may not be detected during plugin registration.
- **ACL EP removed:** Following its deprecation in 1.31.0, the Arm
Compute Library (`ACLExecutionProvider`) execution provider, its build
options, and its standalone C registration API have been removed.
Python's `get_all_providers()` no longer lists `ACLExecutionProvider`;
applications should remove it from explicit provider lists
([#33851](microsoft/onnxruntime#33851)).
- **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:**
In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have
been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP
registered via `OrtApi::RegisterExecutionProviderLibrary`.
`NvTensorRtRtxProviderOptions` is retained as a deprecated declaration
for source compatibility, and legacy `AppendExecutionProvider` calls now
report a deprecation error directing callers to the plugin EP
([#32276](microsoft/onnxruntime#32276)).
- **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**,
raising supported opset ceilings to **28** and restoring
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings
and `Cast` conversions
([#33134](microsoft/onnxruntime#33134)).
- **DLPack v1.3 boolean contract change:** DLPack has been upgraded to
**v1.3** with native `kDLBool` metadata and bidirectional support for
all implemented ORT numeric and boolean types, including all five
`float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2`
(`float6` remains unsupported in DLPack). The low-level `is_bool_tensor`
override was removed, and `uint8` DLPack metadata now always imports as
`uint8`
([#33848](microsoft/onnxruntime#33848)).
- **Weightless compilation API migration:**
`OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the
`ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session
options are deprecated in favor of
`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`,
`"2"` = `ALL`). EPs can now advertise multi-mode capability via
`OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and
`weightless_supported_modes`
([#32899](microsoft/onnxruntime#32899)).
- **Training CI retirement:** Training build and test lanes were retired
from active GitHub Actions CI workflows while retaining inference build
coverage
([#33836](microsoft/onnxruntime#33836)).
- This is the last release for [CocoaPods](https://cocoapods.org/), and
we will switch to [Swift Package
Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/).
- We will use combo onnxruntime-gpu python package to reduce storage
size from next release. You can test it
[here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview)
## New Features
### Core APIs & Runtime
- Added `OrtApi::EpDevice_GetMemoryCapacity` (C++
`EpDevice::GetMemoryCapacity()`) and
`OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total
memory and estimated currently available memory (`DEFAULT` or
`HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support
([#33447](microsoft/onnxruntime#33447)).
- Expanded the app–EP weightless contract so EPs advertise all supported
weightless modes (`weightless_supported_modes`), applications request an
explicit mode (`ModelCompilationOptions_SetWeightlessMode` /
`ep.enable_weightless_mode`), and compiled EPContext models record
`weightless_mode` and `external_dependencies` (`source_model`,
`external_weights`, `ep_context_binary`) in model metadata
([#32899](microsoft/onnxruntime#32899)).
- Extended `session.use_external_initializer_file_buffers_directly="1"`
to `AddExternalInitializers` for zero-copy borrowing of naturally
aligned CPU buffers larger than 127 bytes on little-endian hosts, and
documented the read-only guarantee for external initializers added via
`OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true`
([#33877](microsoft/onnxruntime#33877),
[#33717](microsoft/onnxruntime#33717)).
- Added lock-free multi-worker pinned-memory staging
(`session.external_data_loader.*`) to accelerate external initializer
loading into GPU memory across built-in and plugin CUDA EPs
([#33839](microsoft/onnxruntime#33839)).
- Added a session-scoped MoE/QMoE CPU/GPU offload planner and
memory-budget ledger (`session.moe.offload.*`), generic
`session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`)
configuration plumbing, Level-1 workspace/prepack memory accounting
fallback for nodes absent from memory-profile CSVs, and internal
capability-planning protocol types
([#33451](microsoft/onnxruntime#33451),
[#33042](microsoft/onnxruntime#33042),
[#33311](microsoft/onnxruntime#33311),
[#33841](microsoft/onnxruntime#33841)).
- Enabled cross-thread reuse of idle device streams when graph capture
is disabled and providers register device-selection callbacks, and
enabled compile-session profiling (`<profile_file>_compile`) in
`onnxruntime_perf_test` when `-p` is used with `--compile_ep_context`
([#33053](microsoft/onnxruntime#33053),
[#33694](microsoft/onnxruntime#33694)).
- Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable
without `onnxruntime_MINIMAL_BUILD` and added the
`onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream
support in non-minimal builds
([#33120](microsoft/onnxruntime#33120),
[#33482](microsoft/onnxruntime#33482)).
### ONNX Opset 28 & Shared Contrib Operators
These updates affect core ONNX and contributed operator schemas and
cross-provider behavior, not only the separately released GPU plugins.
- Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and
`CRD`) on CPU and CUDA with version-split registrations across DML, JS,
WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and
DML; opset-28 `BFloat16` support for `Compress`, `OneHot`,
`ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus
`Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16`
`Pad` (opsets 21–25) on CPU and CUDA
([#32362](microsoft/onnxruntime#32362),
[#32369](microsoft/onnxruntime#32369),
[#32378](microsoft/onnxruntime#32378),
[#34062](microsoft/onnxruntime#34062)).
- Extended `GatedDeltaNet` with Kimi Delta Attention (KDA)
per-key-dimension decay across CPU, CUDA, and WebGPU, and extended
`GatedDeltaNet` and `CausalConvWithState` state/gate handling for
multi-token speculative decoding
([#33817](microsoft/onnxruntime#33817),
[#33471](microsoft/onnxruntime#33471)).
- Added an optional `query_seqlens` input to `GroupQueryAttention` to
skip right-padded prefill tokens and idle batch entries
(`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` /
`k_norm_epsilon` attributes to preserve distinct Q and K normalization
epsilons during GQA pre-norm fusion
([#33503](microsoft/onnxruntime#33503),
[#33505](microsoft/onnxruntime#33505),
[#33832](microsoft/onnxruntime#33832),
[#33363](microsoft/onnxruntime#33363)).
- Defined the INT3 `QMoE` v1 packed-code contract, added opt-in
post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU`
(`activation_type="geglu"`) support to MoE/QMoE schemas, and added
`GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt
`global_scale`), and `MoEPacked` (CUDA) to consume native
GGUF/block-packed quantized weights directly
([#33394](microsoft/onnxruntime#33394),
[#33954](microsoft/onnxruntime#33954),
[#33633](microsoft/onnxruntime#33633),
[#33953](microsoft/onnxruntime#33953),
[#33962](microsoft/onnxruntime#33962)).
- Added `PackedSparseAttentionIndexerMerge` for multi-token prediction
index sharing and optional `state_status` / mid-state outputs on
`PackedSparseAttentionIndexer` across CUDA and WebGPU; extended
`GatedAdd` with an optional projected `gate_weight` across CPU, CUDA,
and WebGPU; and supported empty unpacked weights in CPU and WebGPU
`MatMulNBits`
([#33224](microsoft/onnxruntime#33224),
[#33383](microsoft/onnxruntime#33383),
[#33472](microsoft/onnxruntime#33472),
[#33470](microsoft/onnxruntime#33470),
[#33559](microsoft/onnxruntime#33559)).
- Registered core contrib schemas for provider-targeted LLM and
speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`,
`HyperConnectionLowRankMix`, `RouterGateUpProj`,
`SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`,
`ConvStateReplay`, and `MatMulNBitsQLoRA`
([#32126](microsoft/onnxruntime#32126),
[#33469](microsoft/onnxruntime#33469),
[#33384](microsoft/onnxruntime#33384),
[#33650](microsoft/onnxruntime#33650),
[#33474](microsoft/onnxruntime#33474),
[#33213](microsoft/onnxruntime#33213),
[#33493](microsoft/onnxruntime#33493),
[#33370](microsoft/onnxruntime#33370)).
### Plugin Execution Provider Infrastructure
- Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP
boundary by converting `OrtMemoryInfo` instances exclusively through the
C API ([#33571](microsoft/onnxruntime#33571)).
- Zero-initialized `OrtEpFactory` structs in provider-bridge EP
factories so unimplemented optional callbacks are null, and prevented
double DLL initialization and handle leaks when probing and registering
plugin EP libraries
([#33117](microsoft/onnxruntime#33117),
[#33701](microsoft/onnxruntime#33701)).
- Hardened portable model-package EP recreation, custom-domain
preservation, UTF-8 path handling, and `shared_assets` path confinement
and symlink validation
([#32790](microsoft/onnxruntime#32790),
[#33826](microsoft/onnxruntime#33826)).
... (truncated)
Commits viewable in [compare
view](microsoft/onnxruntime@v1.31.1...v1.32.0).
</details>
Updated
[Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime)
from 1.31.1 to 1.32.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.ML.OnnxRuntime.Managed's
releases](https://github.com/microsoft/onnxruntime/releases)._
## 1.32.0
ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support,
introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE
kernels, expands device-memory, weightless-compilation, and Python
model-package APIs, dramatically accelerates session initialization on
large graphs, and strengthens runtime and optimizer reliability. These
notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel
updates are summarized here; detailed provider notes are covered by
their separate plugin EP releases.
## Highlights
- Reintegrated **ONNX 1.23.2** with **opset 28** operator support
(`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions)
and updated **DLPack to v1.3** across all implemented ORT numeric and
boolean tensor types
([#33134](microsoft/onnxruntime#33134),
[#32362](microsoft/onnxruntime#32362),
[#33848](microsoft/onnxruntime#33848)).
- Added a session-scoped **CPU/GPU hybrid MoE offload** planner and
execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and
block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2
CPU `QMoE` kernels and memory-mapped expert storage
([#33450](microsoft/onnxruntime#33450),
[#33451](microsoft/onnxruntime#33451),
[#33452](microsoft/onnxruntime#33452),
[#33973](microsoft/onnxruntime#33973)).
- Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit
multi-mode weightless compilation
(`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode`), Python bindings for the Model Package
APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers`
([#33447](microsoft/onnxruntime#33447),
[#32899](microsoft/onnxruntime#32899),
[#33640](microsoft/onnxruntime#33640),
[#33877](microsoft/onnxruntime#33877)).
- Eliminated superlinear bottlenecks in allocation planning, constant
sharing, graph I/O mapping, and topological sorting—reducing session
initialization on 100k-node constant-heavy graphs from minutes to
seconds—and added pipelined pinned-memory staging for external GPU
weight loading
([#33407](microsoft/onnxruntime#33407),
[#33939](microsoft/onnxruntime#33939),
[#34009](microsoft/onnxruntime#34009),
[#33839](microsoft/onnxruntime#33839)).
- Expanded shared generative AI and quantized contrib operators,
including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension
decay), right-padded chunked prefill in `GroupQueryAttention`, native
GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`,
`MoEPacked`), and sparse-attention index sharing
([#33817](microsoft/onnxruntime#33817),
[#33503](microsoft/onnxruntime#33503),
[#33953](microsoft/onnxruntime#33953),
[#33224](microsoft/onnxruntime#33224)).
## Announcements & Compatibility
- **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched
with ONNX Runtime 1.32.0
([#33214](microsoft/onnxruntime#33214)). For
**contrib-op compatibility**, core and the plugin must use the same
contributed-operator schemas; building both from the **same ONNX Runtime
revision** is recommended. Schema mismatches can cause incorrect
execution or crashes and may not be detected during plugin registration.
- **ACL EP removed:** Following its deprecation in 1.31.0, the Arm
Compute Library (`ACLExecutionProvider`) execution provider, its build
options, and its standalone C registration API have been removed.
Python's `get_all_providers()` no longer lists `ACLExecutionProvider`;
applications should remove it from explicit provider lists
([#33851](microsoft/onnxruntime#33851)).
- **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:**
In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have
been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP
registered via `OrtApi::RegisterExecutionProviderLibrary`.
`NvTensorRtRtxProviderOptions` is retained as a deprecated declaration
for source compatibility, and legacy `AppendExecutionProvider` calls now
report a deprecation error directing callers to the plugin EP
([#32276](microsoft/onnxruntime#32276)).
- **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**,
raising supported opset ceilings to **28** and restoring
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and
`ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings
and `Cast` conversions
([#33134](microsoft/onnxruntime#33134)).
- **DLPack v1.3 boolean contract change:** DLPack has been upgraded to
**v1.3** with native `kDLBool` metadata and bidirectional support for
all implemented ORT numeric and boolean types, including all five
`float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2`
(`float6` remains unsupported in DLPack). The low-level `is_bool_tensor`
override was removed, and `uint8` DLPack metadata now always imports as
`uint8`
([#33848](microsoft/onnxruntime#33848)).
- **Weightless compilation API migration:**
`OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the
`ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session
options are deprecated in favor of
`ModelCompilationOptions_SetWeightlessMode` and
`ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`,
`"2"` = `ALL`). EPs can now advertise multi-mode capability via
`OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and
`weightless_supported_modes`
([#32899](microsoft/onnxruntime#32899)).
- **Training CI retirement:** Training build and test lanes were retired
from active GitHub Actions CI workflows while retaining inference build
coverage
([#33836](microsoft/onnxruntime#33836)).
- This is the last release for [CocoaPods](https://cocoapods.org/), and
we will switch to [Swift Package
Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/).
- We will use combo onnxruntime-gpu python package to reduce storage
size from next release. You can test it
[here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview)
## New Features
### Core APIs & Runtime
- Added `OrtApi::EpDevice_GetMemoryCapacity` (C++
`EpDevice::GetMemoryCapacity()`) and
`OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total
memory and estimated currently available memory (`DEFAULT` or
`HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support
([#33447](microsoft/onnxruntime#33447)).
- Expanded the app–EP weightless contract so EPs advertise all supported
weightless modes (`weightless_supported_modes`), applications request an
explicit mode (`ModelCompilationOptions_SetWeightlessMode` /
`ep.enable_weightless_mode`), and compiled EPContext models record
`weightless_mode` and `external_dependencies` (`source_model`,
`external_weights`, `ep_context_binary`) in model metadata
([#32899](microsoft/onnxruntime#32899)).
- Extended `session.use_external_initializer_file_buffers_directly="1"`
to `AddExternalInitializers` for zero-copy borrowing of naturally
aligned CPU buffers larger than 127 bytes on little-endian hosts, and
documented the read-only guarantee for external initializers added via
`OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true`
([#33877](microsoft/onnxruntime#33877),
[#33717](microsoft/onnxruntime#33717)).
- Added lock-free multi-worker pinned-memory staging
(`session.external_data_loader.*`) to accelerate external initializer
loading into GPU memory across built-in and plugin CUDA EPs
([#33839](microsoft/onnxruntime#33839)).
- Added a session-scoped MoE/QMoE CPU/GPU offload planner and
memory-budget ledger (`session.moe.offload.*`), generic
`session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`)
configuration plumbing, Level-1 workspace/prepack memory accounting
fallback for nodes absent from memory-profile CSVs, and internal
capability-planning protocol types
([#33451](microsoft/onnxruntime#33451),
[#33042](microsoft/onnxruntime#33042),
[#33311](microsoft/onnxruntime#33311),
[#33841](microsoft/onnxruntime#33841)).
- Enabled cross-thread reuse of idle device streams when graph capture
is disabled and providers register device-selection callbacks, and
enabled compile-session profiling (`<profile_file>_compile`) in
`onnxruntime_perf_test` when `-p` is used with `--compile_ep_context`
([#33053](microsoft/onnxruntime#33053),
[#33694](microsoft/onnxruntime#33694)).
- Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable
without `onnxruntime_MINIMAL_BUILD` and added the
`onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream
support in non-minimal builds
([#33120](microsoft/onnxruntime#33120),
[#33482](microsoft/onnxruntime#33482)).
### ONNX Opset 28 & Shared Contrib Operators
These updates affect core ONNX and contributed operator schemas and
cross-provider behavior, not only the separately released GPU plugins.
- Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and
`CRD`) on CPU and CUDA with version-split registrations across DML, JS,
WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and
DML; opset-28 `BFloat16` support for `Compress`, `OneHot`,
`ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus
`Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16`
`Pad` (opsets 21–25) on CPU and CUDA
([#32362](microsoft/onnxruntime#32362),
[#32369](microsoft/onnxruntime#32369),
[#32378](microsoft/onnxruntime#32378),
[#34062](microsoft/onnxruntime#34062)).
- Extended `GatedDeltaNet` with Kimi Delta Attention (KDA)
per-key-dimension decay across CPU, CUDA, and WebGPU, and extended
`GatedDeltaNet` and `CausalConvWithState` state/gate handling for
multi-token speculative decoding
([#33817](microsoft/onnxruntime#33817),
[#33471](microsoft/onnxruntime#33471)).
- Added an optional `query_seqlens` input to `GroupQueryAttention` to
skip right-padded prefill tokens and idle batch entries
(`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` /
`k_norm_epsilon` attributes to preserve distinct Q and K normalization
epsilons during GQA pre-norm fusion
([#33503](microsoft/onnxruntime#33503),
[#33505](microsoft/onnxruntime#33505),
[#33832](microsoft/onnxruntime#33832),
[#33363](microsoft/onnxruntime#33363)).
- Defined the INT3 `QMoE` v1 packed-code contract, added opt-in
post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU`
(`activation_type="geglu"`) support to MoE/QMoE schemas, and added
`GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt
`global_scale`), and `MoEPacked` (CUDA) to consume native
GGUF/block-packed quantized weights directly
([#33394](microsoft/onnxruntime#33394),
[#33954](microsoft/onnxruntime#33954),
[#33633](microsoft/onnxruntime#33633),
[#33953](microsoft/onnxruntime#33953),
[#33962](microsoft/onnxruntime#33962)).
- Added `PackedSparseAttentionIndexerMerge` for multi-token prediction
index sharing and optional `state_status` / mid-state outputs on
`PackedSparseAttentionIndexer` across CUDA and WebGPU; extended
`GatedAdd` with an optional projected `gate_weight` across CPU, CUDA,
and WebGPU; and supported empty unpacked weights in CPU and WebGPU
`MatMulNBits`
([#33224](microsoft/onnxruntime#33224),
[#33383](microsoft/onnxruntime#33383),
[#33472](microsoft/onnxruntime#33472),
[#33470](microsoft/onnxruntime#33470),
[#33559](microsoft/onnxruntime#33559)).
- Registered core contrib schemas for provider-targeted LLM and
speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`,
`HyperConnectionLowRankMix`, `RouterGateUpProj`,
`SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`,
`ConvStateReplay`, and `MatMulNBitsQLoRA`
([#32126](microsoft/onnxruntime#32126),
[#33469](microsoft/onnxruntime#33469),
[#33384](microsoft/onnxruntime#33384),
[#33650](microsoft/onnxruntime#33650),
[#33474](microsoft/onnxruntime#33474),
[#33213](microsoft/onnxruntime#33213),
[#33493](microsoft/onnxruntime#33493),
[#33370](microsoft/onnxruntime#33370)).
### Plugin Execution Provider Infrastructure
- Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP
boundary by converting `OrtMemoryInfo` instances exclusively through the
C API ([#33571](microsoft/onnxruntime#33571)).
- Zero-initialized `OrtEpFactory` structs in provider-bridge EP
factories so unimplemented optional callbacks are null, and prevented
double DLL initialization and handle leaks when probing and registering
plugin EP libraries
([#33117](microsoft/onnxruntime#33117),
[#33701](microsoft/onnxruntime#33701)).
- Hardened portable model-package EP recreation, custom-domain
preservation, UTF-8 path handling, and `shared_assets` path confinement
and symlink validation
([#32790](microsoft/onnxruntime#32790),
[#33826](microsoft/onnxruntime#33826)).
... (truncated)
Commits viewable in [compare
view](microsoft/onnxruntime@v1.31.1...v1.32.0).
</details>
Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest)
from 18.11.0 to 18.12.0.
<details>
<summary>Release notes</summary>
_Sourced from [Microsoft.NET.Test.Sdk's
releases](https://github.com/microsoft/vstest/releases)._
## 18.12.0
## Security fix
This release includes the fix for CVE-2026-105544, an elevation of
privilege vulnerability in the VSTest artifact post-processor. The
post-processor could consume a forged `executionComplete.json` written
by another local user, and then run attacker-controlled output handling.
Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every
version from 18.11.1 on is fixed, including 18.12.0.
`Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the
.NET SDK. The recommended way to get this fix is to update your .NET SDK
to the latest patched version, see the advisory for the SDK versions.
The fix is also released in the `Microsoft.TestPlatform.CLI` package,
for completeness. That package is rarely used directly. It is not a
dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so
those packages do not bring it in.
Details: microsoft/vstest#16654 and
dotnet/announcements#517
Using custom Docker images or Nano Server? See
microsoft/vstest#16652
## What's Changed
* Fix TestCaseStart/Stop events dropped for data-driven tests with
shared TestCase.Id by @nohwnd in
microsoft/vstest#16033
* Stop the discovery manager factory from shadowing discoveryCriteria by
@nohwnd in microsoft/vstest#16391
* Gate the HTTP link checker on a fingerprint of the broken links by
@nohwnd in microsoft/vstest#16413
* Quote boolean conditions in shipped Microsoft.TestPlatform.targets by
@nohwnd in microsoft/vstest#16404
* Obsolete API hygiene: the compatibility-safe changes by @azat-msft in
microsoft/vstest#16420
* Add xxHash128 test ids, available but not default by @azat-msft in
microsoft/vstest#16378
* Isolate attachment manager test temporary files by @mmitche in
microsoft/vstest#16463
* Test ids report logger by @azat-msft in
microsoft/vstest#16443
* Isolate code coverage files in published output by @azat-msft in
microsoft/vstest#16483
* Preserve OutputType for Android test projects by @jonathanpeppers in
microsoft/vstest#16496
* Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft
in microsoft/vstest#16462
* Mark the xxHash128 test id API as experimental by @azat-msft in
microsoft/vstest#16465
* Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform
and Portable packages by @nohwnd in
microsoft/vstest#16633
* Fix localized resource paths in CLI source-build/non-Windows packages
by @nohwnd in microsoft/vstest#16644
**Full Changelog**:
microsoft/vstest@v18.11.0...v18.12.0
## 18.11.1
## Security fix
This release fixes CVE-2026-105544, an elevation of privilege
vulnerability in the VSTest artifact post-processor. The post-processor
could consume a forged `executionComplete.json` written by another local
user, and then run attacker-controlled output handling.
Affected: `Microsoft.TestPlatform.CLI` 18.11.0.
Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1.
`Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the
.NET SDK. The recommended way to get this fix is to update your .NET SDK
to the latest patched version, see the advisory for the SDK versions.
The fix is also released in the `Microsoft.TestPlatform.CLI` package,
for completeness. That package is rarely used directly. It is not a
dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so
those packages do not bring it in.
Details: microsoft/vstest#16654 and
dotnet/announcements#517
Using custom Docker images or Nano Server? See
microsoft/vstest#16652
## What's Changed
* Security fix for CVE-2026-105544, see
microsoft/vstest#16654
* Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform
and Portable packages by @nohwnd in
microsoft/vstest#16632
* Fix localized resource paths in CLI source-build packages by @nohwnd
in microsoft/vstest#16643
**Full Changelog**:
microsoft/vstest@v18.11.0...v18.11.1
Commits viewable in [compare
view](microsoft/vstest@v18.11.0...v18.12.0).
</details>
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR accelerates external initializer loading to GPU memory for both the built-in CUDA Execution Provider and the CUDA Plugin Execution Provider without introducing external storage dependencies (
libcufile/ GPUDirect Storage or DirectStorage):cudaStreamSynchronizebarrier with a shared lock-free work queue (ExecuteExternalDataLoadTasksinonnxruntime/core/framework/external_data_loader_thread_pool.h).2 MiBdefault) via atomic index and overlapping multi-threaded file reads (preadon POSIX /OVERLAPPEDReadFileon Windows) withcudaMemcpyAsyncacross tensor boundaries.PluginExecutionProvider) Support:PluginExecutionProvider::GetExternalDataLoader()usinghost_accessible_memory_info(Reserve/Freefor pinned memory), lightweight copy-onlyOrtSyncStreamImplstreams (copy_only=1, skipping unnecessary cuBLAS/cuDNN handle creation), andOrtDataTransferImpl(CopyTensorAsync).CudaDataTransfer::CopyTensorsImplandCudaSyncStream::FlushImplselect and restore the target CUDA device when invoked from reader worker threads.IExternalDataLoaderandExternalDataLoaderManagerwithBeginLoad(),FinalizeLoad(), andEndLoad()so external tensors inSaveInitializedTensorsare pipelined across tensor boundaries and synchronized once inFinalizeLoad().SaveInitializedTensorsby(external_location, external_offset, ort_value_index)for sequential file access instead of hash-bucket order, and caches validatedRandomAccessFilehandles perSaveInitializedTensorscall.clamp(physical_cores / 2, 2, 8)).16 MiBbuffers, 4x smaller than the previous 128 MiB default).FinalizeLoad()and gates it on a 16 MiB minimum total external initializer size on the device (session.external_data_loader.min_size_bytes), so small models incur zero pinned-memory or stream-creation overhead.session.external_data_loader.threads,session.external_data_loader.pinned_memory_bytes,session.external_data_loader.chunk_bytes,session.external_data_loader.min_size_bytes) alongside matching CUDA EP provider options (external_data_loader_reading_threads,external_data_loader_pinned_memory_bytes,external_data_loader_chunk_bytes).!defined(ORT_MINIMAL_BUILD)and!defined(USE_CUDA_MINIMAL)).Summary of Changes
include/onnxruntime/core/session/onnxruntime_session_options_config_keys.hsession.external_data_loader.{threads,pinned_memory_bytes,chunk_bytes,min_size_bytes}keysinclude/onnxruntime/core/providers/cuda/cuda_provider_options.hexternal_data_loader_{pinned_memory_bytes,chunk_bytes}and conservative defaults toOrtCUDAProviderOptionsV2onnxruntime/core/framework/external_data_loader.hExternalDataLoaderOptions, validation/parsing helpers, andConfigure/BeginLoad/FinalizeLoad/EndLoadhooksonnxruntime/core/framework/external_data_loader_thread_pool.hstd::threadworker pool and lock-free double-bufferedExecuteExternalDataLoadTaskspipelineonnxruntime/core/framework/external_data_loader_manager.{h,cc}BeginLoad,FinalizeLoad, andEndLoadbatch lifecycle managementonnxruntime/core/framework/session_state_utils.cc(location, offset), cache validated file handles, and bracket loading withBeginLoad/FinalizeLoad/EndLoadonnxruntime/core/framework/tensorprotoutils.{h,cc}ExternalDataFileCacheto validate and open each external.datafile once perSaveInitializedTensorspassonnxruntime/core/providers/cuda/cuda_external_data_loader.{h,cc}onnxruntime/core/providers/cuda/cuda_execution_provider_info.{h,cc},cuda_provider_factory.cc,cuda_execution_provider.cc,provider_bridge_ort.cconnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.{h,cc},onnxruntime/core/providers/cuda/plugin/*PluginExternalDataLoader, copy-only sync streams, device-safe worker H2D copies, and option validationdocs/Model_Loading_Performance.mdonnxruntime/test/framework/external_data_loader_test.cc,cuda_external_data_loader_test.cc,test_cuda_plugin_ep.pyMotivation and Context
Loading large ONNX models with external weights (e.g.,
qwen_flash_next_int4_int8_rc5_contrib_dflash2/model.onnx, 68.46 GiB across 899 external tensors in 7.datafiles) into GPU memory duringInferenceSessioncreation had several bottlenecks:PluginExecutionProvider(used when building the CUDA EP as a plugin) did not implementGetExternalDataLoader(), falling back to per-tensormmap+ single-threaded pageablecudaMemcpy.nvidia-fsis unavailable), a global thread-pool barrier per 64 MiB chunk and acudaStreamSynchronizebarrier at the end of everyLoadTensor()call prevented CPU/GPU overlap across tensors<= 64 MiB(705 of 899 tensors), forced all 613 tensors< 16 MiBonto a single thread, iterated over initializers in arbitrary hash-bucket order, and reopened/revalidated.datafiles for every tensor.Benchmark Comparison (
qwen_flash_next_int4_int8_rc5_contrib_dflash2/model.onnx, 68.46 GiB, 899 External Tensors, NVIDIA H200)InferenceSessionTimeorigin/mainCUDA Plugin EP)use_gds=1-> 4-thread barrier fallback)Even at the exact same 4-thread count and 4x smaller pinned memory (32 MiB vs 128 MiB), the lock-free cross-tensor pipeline + offset sorting + file-handle caching reduces session creation time from 5.110 s to 3.730 s (1.37x faster). With the CPU-core-derived default (
8threads) and conservative32 MiBpinned memory, session creation completes in 2.191 s (31.25 GiB/s).Validation
./onnxruntime_test_all --gtest_filter="*ExternalDataLoader*:*Initializer*"— 127 tests ran, 126 passed, 1 skipped (32-bit platform check), 0 failed.ORT_TEST_CUDA_PLUGIN_EP=1 python onnxruntime/test/python/transformers/test_cuda_plugin_ep.py— 94 tests ran, 92 passed, 2 skipped, 0 failed (includingtest_external_data_loader_modes_and_validation).lintrunner -a— passed (0 lint issues).