[CUDA] Add session-scoped MoE/QMoE offload planner, lifecycle controller, and memory budget ledger by tianleiwu · Pull Request #33451 · microsoft/onnxruntime

Tianlei Wu (tianleiwu) · 2026-10-19T08:45:14Z

Description

Stacked on #33450, structured into 2 modular commits:

Commit 1 (f9507fa74b) — Session-Scoped MoE/QMoE Offload Planner & Device/Pinned Memory Budget Ledger:
- Adds session.moe.offload.* configuration keys, MoeOffloadSessionPolicy, MoeNodeExecutionPlan, and the 5-stage deterministic admission planner (moe_offload_plan.h).
- Adds Stage 1 PlaceMoeOffloadInitializersOnCpu in SessionState (session_state.cc) and CudaDeviceMemoryBudget / BoundedPinnedPool across CUDA EP and Plugin EP allocators/arenas.
Commit 2 (a4df250105) — QMoESessionController & CUDA EP / Plugin EP Lifecycle Hooks:
- Adds QMoESessionController (qmoe_session_controller.h) for session-scoped MoE offload admission, shared landing ring management, Linux read-only file-mapping promotion/eviction, and run/graph-replay synchronization.
- Wires OnSessionInitializationEnd, OnRunStart, OnRunEnd, and CUDA Graph capture/replay generation tracking in CUDAExecutionProvider and CudaEp (Plugin EP).

Documentation

Added docs/moe-cpu-offload/qmoe_offload.md documenting the session.moe.offload.* configuration keys, external data layout & PromoteFileMappingToSharedReadOnly, 5-stage SessionState planning & admission pipeline, CUDA memory budget ledgers (CudaDeviceMemoryBudget, SessionBudgetAccount, HostPinnedBudgetLedger), and QMoESessionController lifecycle hooks.

Copilot

🟡 Changes recommended

Unresolved initializer-placement, graph-memory lifetime, budget-accounting, and lifecycle-cleanup defects block approval.

12 open issues

Propagate auto-mode fallback through shared initializers · New
Gate CPU placement until offload kernels are available · New
Make controller lifecycle cleanup exception-safe · New
Make plugin controller cleanup exception-safe · New
Bind allocators to explicit session identities · New
Protect graph-retained regions from cross-session shrinking · New
Populate ranked experts for full GPU residency · New
Reserve dummy slots before distributing resident capacity · New
Enforce per-node host expert byte limits · New
Reclaim idle blocks before rejecting larger allocations · New
Invoke after-replay hook after capture · New
Apply node caps to resident, spare, and dummy slots · New

What changed in this PR

Adds session-level CUDA MoE/QMoE offload planning and resource management, building on the CPU expert-storage support from #33450.

Changes:

Introduces offload configuration, initializer placement, and residency planning.
Adds device/session budget accounting and reusable pinned buffers.
Integrates controller lifecycle, graph replay, allocator statistics, and plugin arena shrinking.

File	Description
`onnxruntime/core/session/plugin_ep/ep_plugin_provider_interfaces.cc`	Forwards CUDA allocator options.
`onnxruntime/core/session/inference_session.cc`	Adds scoped provider cleanup and corrects shrink validation.
`onnxruntime/core/session/allocator_adapters.h`	Exposes wrapped allocator overrides.
`onnxruntime/core/session/allocator_adapters.cc`	Forwards wrapped allocator statistics.
`onnxruntime/core/providers/cuda/qmoe_session_controller.h`	Adds planning, residency management, and invocation lifecycle.
`onnxruntime/core/providers/cuda/plugin/cuda_kernel_adapter.h`	Exposes controller and pinned-buffer helpers.
`onnxruntime/core/providers/cuda/plugin/cuda_graph_plugin.h`	Tracks graph recapture state.
`onnxruntime/core/providers/cuda/plugin/cuda_graph_plugin.cc`	Updates recapture and asynchronous replay handling.
`onnxruntime/core/providers/cuda/plugin/cuda_ep.h`	Declares budget and initialization integration.
`onnxruntime/core/providers/cuda/plugin/cuda_ep.cc`	Integrates controller lifecycle and replay hooks.
`onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.h`	Declares account tracking and arena shrinking.
`onnxruntime/core/providers/cuda/plugin/cuda_ep_factory.cc`	Creates budget accounts and connects allocators.
`onnxruntime/core/providers/cuda/plugin/cuda_data_transfer_plugin.cc`	Adds allocation-failure diagnostics.
`onnxruntime/core/providers/cuda/plugin/cuda_arena.h`	Adds option aliases and shrink eligibility.
`onnxruntime/core/providers/cuda/plugin/cuda_arena.cc`	Exposes budget statistics and records shrink eligibility.
`onnxruntime/core/providers/cuda/plugin/cuda_allocator_plugin.h`	Declares budget-aware allocator integration.
`onnxruntime/core/providers/cuda/plugin/cuda_allocator_plugin.cc`	Delegates allocation accounting and exposes statistics.
`onnxruntime/core/providers/cuda/moe_offload_plan.h`	Re-exports shared offload definitions.
`onnxruntime/core/providers/cuda/cuda_provider_factory.cc`	Forwards the device-memory limit.
`onnxruntime/core/providers/cuda/cuda_kernel.h`	Adds controller and pinned-buffer accessors.
`onnxruntime/core/providers/cuda/cuda_execution_provider.h`	Declares controller ownership and graph invalidation.
`onnxruntime/core/providers/cuda/cuda_execution_provider.cc`	Integrates budgets, admission, and run lifecycle.
`onnxruntime/core/providers/cuda/cuda_execution_provider_info.h`	Stores and hashes the memory-limit option.
`onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc`	Parses and serializes the memory-limit option.
`onnxruntime/core/providers/cuda/cuda_device_memory_budget.h`	Defines physical, session, group, and pinned-memory ledgers.
`onnxruntime/core/providers/cuda/cuda_allocator.h`	Declares allocation tracking and budget access.
`onnxruntime/core/providers/cuda/cuda_allocator.cc`	Accounts for reservations, frees, and quarantine.
`onnxruntime/core/providers/cuda/bounded_pinned_pool.h`	Adds bounded pinned-buffer reuse.
`onnxruntime/core/framework/session_state.cc`	Validates offload eligibility and relocates initializers.
`onnxruntime/core/framework/session_state_utils.cc`	Skips tracing external CPU initializers.
`onnxruntime/core/framework/moe_offload_plan.h`	Defines policy parsing and backend planning contracts.
`include/onnxruntime/ep/adapter/op_kernel.h`	Exposes parallel execution through the adapter.
`include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h`	Defines offload configuration keys.
`include/onnxruntime/core/providers/cuda/cuda_provider_options.h`	Adds the device-memory-limit field.
`include/onnxruntime/core/framework/allocator.h`	Adds wrapped allocator access.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Unresolved allocation-budget enforcement, lifecycle cleanup, and admission defects prevent reliable operation.

7 open issues

Use retryable failure for budget exhaustion · New
Wait for CPU completion before releasing run · New
Construct cleanup guard before provider startup · New
Share budget account with plugin temp allocator · New
Reject overrides below mandatory base cost · New
Allow direct full-residency transition when affordable · New
Honor minimum host cache slots in bounded cache · New

12 resolved since last review

_{Review effort: Balanced}

…ntry gpu_device_mem_limit was added as a member of OrtCUDAProviderOptionsV2, which is public API, right next to the similarly named gpu_mem_limit. Move it to the session config entry session.moe.offload.gpu_device_mem_limit so the public struct is not extended for it and the two limits are no longer easy to confuse. The CUDA EP and the CUDA plugin EP both read the same key. - Add kOrtSessionOptionsMoeOffloadGpuDeviceMemLimit and parse it in MoeOffloadSessionPolicy, so a malformed value is rejected with the key name for every session, including ones that do not use the CUDA EP. - CUDA EP: CUDAProviderFactory::CreateProvider(session_options, logger) reads the key and fills CUDAExecutionProviderInfo::gpu_device_mem_limit. All registration paths (C API V1 and V2, Python, automatic EP selection) already call this overload. - Remove the member from OrtCUDAProviderOptionsV2 and the "gpu_device_mem_limit" provider option with its parse/serialize entries; it is now an unknown provider option. CUDAExecutionProviderInfo keeps the field because it is internal and takes part in the provider-info hash. - CUDA plugin EP: CreateEp reads the session-level key instead of the EP-scoped option. For the plugin named CUDAExecutionProvider the host forwards it as the gpu_device_mem_limit allocator option, so allocator requests that no pending session account owns still get the limit. The old mapping of ep.cudaexecutionprovider.gpu_device_mem_limit is removed. - Name the session key in the "invalid value", "external CUDA allocator" and "CUDA memory pool" errors. - Update qmoe_offload.md. min_num_runs_before_cuda_graph_capture stays in OrtCUDAProviderOptionsV2, as requested in review. Add tests for the key parsing, the host forwarding to the plugin allocator factory, the in-tree factory route (C API V1 and V2 options), the plugin EP, the rejected provider-option spelling and the invalid-value and external-allocator errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tianlei Wu (tianleiwu) · 2026-10-31T00:32:34Z

Follow-up on the gpu_device_mem_limit / ABI discussion, pushed as a separate commit (89278e9).

gpu_device_mem_limit is no longer a member of OrtCUDAProviderOptionsV2. It is now the session config entry session.moe.offload.gpu_device_mem_limit (bytes; 0 or unset = no limit), defined next to the other offload keys and documented in qmoe_offload.md. Both CUDA EPs honor it, and it has to be set before the session is created.

In-tree CUDA EP: CUDAProviderFactory::CreateProvider(session_options, logger) reads the key and fills the internal CUDAExecutionProviderInfo::gpu_device_mem_limit. The C API (V1 and V2 options) and the Python bindings both create the provider through that overload. The gpu_device_mem_limit provider option and its parse/serialize entries are removed, so it is now an unknown provider option.
CUDA plugin EP: CreateEp reads the session key. For the plugin named CUDAExecutionProvider, the host forwards it as the existing gpu_device_mem_limit allocator option, so allocator requests that no pending session account owns still get the limit. The EP-scoped spelling ep.cudaexecutionprovider.gpu_device_mem_limit is no longer mapped.
Malformed values ("", -1, 1.5, 1 GiB, out of range) fail session creation with an error that names the key, also on sessions that do not use the CUDA EP. The limit is still rejected together with external CUDA allocators and CUDA memory pools, and those errors now name the key too.

The member was introduced by this PR, so no released behavior changes.

Not changed: min_num_runs_before_cuda_graph_capture is still stored in OrtCUDAProviderOptionsV2, as requested in the review thread on cuda_execution_provider_info.cc. It has the same ABI consideration. If you would rather keep the public struct untouched for it as well, I can move it in another separate commit.

Tests

CPU build: MoeOffloadSessionPolicyTest.* (key parsing), InferenceSessionTests.MoeOffloadGpuDeviceMemLimit*, and PluginExecutionProviderTest.CreatePreferredAllocators_*DeviceMemLimit* (the host forwards the key only to the CUDA plugin EP and ignores the EP-scoped spelling), plus the existing offload and session-option suites.
In-tree CUDA EP: CApiTest.TestCUDASessionDeviceMemLimit* (V1 and V2 options: a 1 GiB limit runs, a 1 KiB limit fails session creation, invalid values are rejected), TestCUDAProviderOptionsDoNotCarryDeviceMemLimit, CudaProviderInfoTest.GpuDeviceMemLimit*, and the full CUDA unit-test module (551 passed; one flash-attention test is excluded because this build has flash attention off).
CUDA plugin EP: CudaPluginArenaTest.SessionDeviceMemLimit_* plus the account-binding, budget-group, partitioning and GEMM-bias suites.

… logger data out of the CUDA test module Run cleanup (InferenceSession::RunImpl) - Create the exactly-once ProvidersStopGuard before the provider OnRunStart loop instead of after it. A throw from a later provider's OnRunStart used to skip the guard, so providers that had already started the run never received OnRunEnd and the next serial Run on the session failed. The guard now ends the run, as failed, on exactly the providers that started it while the exception unwinds. - Add RunStartFailureTest: a later provider that throws from OnRunStart, and one that returns an error status. The provider that started the run gets one OnRunEnd(failed) and the same session then runs successfully. Windows test module (QMoESessionController) - The header-only controller logged from the inline PlanAndAdmitSession. In the CUDA test module that expands to the core LOGS_DEFAULT, whose inline LoggingManager::DefaultLogger() reads the private static s_default_logger_, so the module referenced host data without __declspec(dllimport) and tools/ci_build/gen_test_exports.py rejected the Windows build. That validation is unchanged. - PlanAndAdmitSession now fills a SessionPlanReport (the WDDM paging warning and the plan summary) and does not log. The built-in EP and the plugin EP log the report from their own translation units, each with its own logger. The built-in EP still logs to the default logger. The plugin EP now logs through the session OrtLogger, like its other diagnostics, instead of the adapter default logger. - cuda_execution_provider_test.cc no longer calls LoggingManager::DefaultLogger() either; it creates a logger from the test environment's logging manager. - Add admission tests for the reported plan: the summary text, whether the bounded file cache is used, and that rejected or already committed plans report nothing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The RunStartFailureTest added in the previous commit grew inference_session_test.cc enough that the Windows Debug and ASAN builds of onnxruntime_test_all fail with C1128 (number of sections exceeded object file format limit). Use the existing per-source /bigobj property that already covers the large optimizer test translation units. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tianlei Wu (tianleiwu) · 2026-10-31T02:53:47Z

CI follow-up: the previous push (0804bd560c) added the run-start failure test to inference_session_test.cc, which pushed that object file over the MSVC section limit in the Windows Debug and ASAN builds (error C1128). 0e56b097cc adds the file to the existing per-source /bigobj list in cmake/onnxruntime_unittests.cmake (the same mechanism already used for the large optimizer test files). It is a build-flag change only, with no code change.

windows_x64_debug and windows_x64_asan pass on this head.

…, adaptive promotion, and DMA prefill (#33452) ### Description Stacked on #33451, rebased onto the updated `main` + #33450 + #33451 stack and structured into 2 commits: 1. **Commit 1 (`18928e7e46`) — QMoE Offload Kernels, SpinPool, Callback Timing, and `QMoE` v4 Admission Hook**: - Adds `LaunchQMoEOffloadClassify`, `LaunchQMoEOffloadGather`, `LaunchQMoEOffloadScatter`, and `LaunchQMoEOffloadMergeFinalizeCombine` in `qmoe_kernels.{h,cu}` and raw NVFP4 GEMV decode support in `moe_gemv_fp4.cu`. - Adds `SpinPool` (`qmoe_spin_pool.h`), callback timing (`qmoe_callback_timing.h`), and v4 `QMoE` offload admission hooks in `moe_quantization.{h,cc}`. 2. **Commit 2 (`9f2abe75e9`) — Portable `QMoEOffload` Backend, Adaptive Promotion, and v4 Offload Tests**: - Adds `QMoEOffload` (`qmoe_offload.{h,cc}`), `test_qmoe_offload_v4.py`, and `docs/contrib_ops/cuda/moe_qmoe.md`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Updated [Microsoft.AspNetCore.Mvc.Testing](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.AspNetCore.Mvc.Testing's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.Data.Sqlite](https://github.com/dotnet/dotnet) from 10.0.14 to 10.0.15. <details> <summary>Release notes</summary> _Sourced from [Microsoft.Data.Sqlite's releases](https://github.com/dotnet/dotnet/releases)._ No release notes found for this version range. Commits viewable in [compare view](https://github.com/dotnet/dotnet/commits). </details> Updated [Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime's releases](https://github.com/Microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.ML.OnnxRuntime.Managed](https://github.com/microsoft/onnxruntime) from 1.31.1 to 1.32.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.ML.OnnxRuntime.Managed's releases](https://github.com/microsoft/onnxruntime/releases)._ ## 1.32.0 ONNX Runtime 1.32.0 reintegrates ONNX 1.23.2 with opset 28 support, introduces CPU/GPU hybrid MoE offloading and shared-layout CPU QMoE kernels, expands device-memory, weightless-compilation, and Python model-package APIs, dramatically accelerates session initialization on large graphs, and strengthens runtime and optimizer reliability. These notes cover changes since ONNX Runtime 1.31.1. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Reintegrated **ONNX 1.23.2** with **opset 28** operator support (`SpaceToDepth`, `Mod`, signed `BitShift`, and `BFloat16` extensions) and updated **DLPack to v1.3** across all implemented ORT numeric and boolean tensor types ([#33134](microsoft/onnxruntime#33134), [#32362](microsoft/onnxruntime#32362), [#33848](microsoft/onnxruntime#33848)). - Added a session-scoped **CPU/GPU hybrid MoE offload** planner and execution backend for quantized `QMoE` (NVFP4, INT4, INT3, and INT2) and block-packed `MoEPacked` models, backed by shared-layout AVX-512/AVX2 CPU `QMoE` kernels and memory-mapped expert storage ([#33450](microsoft/onnxruntime#33450), [#33451](microsoft/onnxruntime#33451), [#33452](microsoft/onnxruntime#33452), [#33973](microsoft/onnxruntime#33973)). - Expanded core APIs with `OrtApi::EpDevice_GetMemoryCapacity`, explicit multi-mode weightless compilation (`ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode`), Python bindings for the Model Package APIs, and zero-copy CPU buffer borrowing in `AddExternalInitializers` ([#33447](microsoft/onnxruntime#33447), [#32899](microsoft/onnxruntime#32899), [#33640](microsoft/onnxruntime#33640), [#33877](microsoft/onnxruntime#33877)). - Eliminated superlinear bottlenecks in allocation planning, constant sharing, graph I/O mapping, and topological sorting—reducing session initialization on 100k-node constant-heavy graphs from minutes to seconds—and added pipelined pinned-memory staging for external GPU weight loading ([#33407](microsoft/onnxruntime#33407), [#33939](microsoft/onnxruntime#33939), [#34009](microsoft/onnxruntime#34009), [#33839](microsoft/onnxruntime#33839)). - Expanded shared generative AI and quantized contrib operators, including Kimi Delta Attention (`GatedDeltaNet` per-key-dimension decay), right-padded chunked prefill in `GroupQueryAttention`, native GGUF/block-packed operators (`GatherPacked`, `MatMulPacked`, `MoEPacked`), and sparse-attention index sharing ([#33817](microsoft/onnxruntime#33817), [#33503](microsoft/onnxruntime#33503), [#33953](microsoft/onnxruntime#33953), [#33224](microsoft/onnxruntime#33224)). ## Announcements & Compatibility - **CUDA Plugin EP v0.4.0** and **WebGPU Plugin EP v0.6.0** are matched with ONNX Runtime 1.32.0 ([#33214](microsoft/onnxruntime#33214)). For **contrib-op compatibility**, core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected during plugin registration. - **ACL EP removed:** Following its deprecation in 1.31.0, the Arm Compute Library (`ACLExecutionProvider`) execution provider, its build options, and its standalone C registration API have been removed. Python's `get_all_providers()` no longer lists `ACLExecutionProvider`; applications should remove it from explicit provider lists ([#33851](microsoft/onnxruntime#33851)). - **Legacy provider-bridge `NvTensorRTRTXExecutionProvider` removed:** In-tree provider-bridge `NvTensorRTRTXExecutionProvider` sources have been removed in favor of the standalone NVIDIA TensorRT RTX plugin EP registered via `OrtApi::RegisterExecutionProviderLibrary`. `NvTensorRtRtxProviderOptions` is retained as a deprecated declaration for source compatibility, and legacy `AppendExecutionProvider` calls now report a deprecation error directing callers to the plugin EP ([#32276](microsoft/onnxruntime#32276)). - **ONNX 1.23.2 integration:** ONNX has been upgraded to **1.23.2**, raising supported opset ceilings to **28** and restoring `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E2M3` and `ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT6E3M2` public element-type mappings and `Cast` conversions ([#33134](microsoft/onnxruntime#33134)). - **DLPack v1.3 boolean contract change:** DLPack has been upgraded to **v1.3** with native `kDLBool` metadata and bidirectional support for all implemented ORT numeric and boolean types, including all five `float8` formats, packed `float4`, `int4`/`uint4`, and `int2`/`uint2` (`float6` remains unsupported in DLPack). The low-level `is_bool_tensor` override was removed, and `uint8` DLPack metadata now always imports as `uint8` ([#33848](microsoft/onnxruntime#33848)). - **Weightless compilation API migration:** `OrtCompileApi::ModelCompilationOptions_SetWeightlessEnabled` and the `ep.enable_weightless` / `ep.enable_weightless_ep_context_nodes` session options are deprecated in favor of `ModelCompilationOptions_SetWeightlessMode` and `ep.enable_weightless_mode` (`"0"` = `NONE`, `"1"` = `EXTERNAL_ONLY`, `"2"` = `ALL`). EPs can now advertise multi-mode capability via `OrtWeightlessSupport_ALL_OR_EXTERNAL_ONLY` and `weightless_supported_modes` ([#32899](microsoft/onnxruntime#32899)). - **Training CI retirement:** Training build and test lanes were retired from active GitHub Actions CI workflows while retaining inference build coverage ([#33836](microsoft/onnxruntime#33836)). - This is the last release for [CocoaPods](https://cocoapods.org/), and we will switch to [Swift Package Manager](https://docs.swift.org/latest/documentation/packagemanagerdocs/). - We will use combo onnxruntime-gpu python package to reduce storage size from next release. You can test it [here](https://aiinfra.visualstudio.com/PublicPackages/_artifacts/feed/ORT-Nightly/PyPI/onnxruntime-gpu/overview) ## New Features ### Core APIs & Runtime - Added `OrtApi::EpDevice_GetMemoryCapacity` (C++ `EpDevice::GetMemoryCapacity()`) and `OrtEpFactory::GetDeviceMemoryCapacity` to query device-global total memory and estimated currently available memory (`DEFAULT` or `HOST_ACCESSIBLE`), with built-in and plugin CUDA EP support ([#33447](microsoft/onnxruntime#33447)). - Expanded the app–EP weightless contract so EPs advertise all supported weightless modes (`weightless_supported_modes`), applications request an explicit mode (`ModelCompilationOptions_SetWeightlessMode` / `ep.enable_weightless_mode`), and compiled EPContext models record `weightless_mode` and `external_dependencies` (`source_model`, `external_weights`, `ep_context_binary`) in model metadata ([#32899](microsoft/onnxruntime#32899)). - Extended `session.use_external_initializer_file_buffers_directly="1"` to `AddExternalInitializers` for zero-copy borrowing of naturally aligned CPU buffers larger than 127 bytes on little-endian hosts, and documented the read-only guarantee for external initializers added via `OrtModelEditorApi::AddInitializerToGraph` with `data_is_external=true` ([#33877](microsoft/onnxruntime#33877), [#33717](microsoft/onnxruntime#33717)). - Added lock-free multi-worker pinned-memory staging (`session.external_data_loader.*`) to accelerate external initializer loading into GPU memory across built-in and plugin CUDA EPs ([#33839](microsoft/onnxruntime#33839)). - Added a session-scoped MoE/QMoE CPU/GPU offload planner and memory-budget ledger (`session.moe.offload.*`), generic `session.kernel_dispatch_policy` (`auto`, `latency`, `memory`, `safe`) configuration plumbing, Level-1 workspace/prepack memory accounting fallback for nodes absent from memory-profile CSVs, and internal capability-planning protocol types ([#33451](microsoft/onnxruntime#33451), [#33042](microsoft/onnxruntime#33042), [#33311](microsoft/onnxruntime#33311), [#33841](microsoft/onnxruntime#33841)). - Enabled cross-thread reuse of idle device streams when graph capture is disabled and providers register device-selection callbacks, and enabled compile-session profiling (`<profile_file>_compile`) in `onnxruntime_perf_test` when `-p` is used with `--compile_ep_context` ([#33053](microsoft/onnxruntime#33053), [#33694](microsoft/onnxruntime#33694)). - Made `onnxruntime_DISABLE_EXCEPTIONS` (`--disable_exceptions`) usable without `onnxruntime_MINIMAL_BUILD` and added the `onnxruntime_DISABLE_STREAM=ON` CMake option to disable device-stream support in non-minimal builds ([#33120](microsoft/onnxruntime#33120), [#33482](microsoft/onnxruntime#33482)). ### ONNX Opset 28 & Shared Contrib Operators These updates affect core ONNX and contributed operator schemas and cross-provider behavior, not only the separately released GPU plugins. - Added ONNX opset 28 operator support: native `SpaceToDepth` (`DCR` and `CRD`) on CPU and CUDA with version-split registrations across DML, JS, WebGPU, and QNN; opset-28 floating floor-modulus `Mod` on CPU, CUDA, and DML; opset-28 `BFloat16` support for `Compress`, `OneHot`, `ReverseSequence`, `Unique`, `Celu`, and `Einsum` on CPU (plus `Compress`, `OneHot`, and `ReverseSequence` on CUDA); and `BFloat16` `Pad` (opsets 21–25) on CPU and CUDA ([#32362](microsoft/onnxruntime#32362), [#32369](microsoft/onnxruntime#32369), [#32378](microsoft/onnxruntime#32378), [#34062](microsoft/onnxruntime#34062)). - Extended `GatedDeltaNet` with Kimi Delta Attention (KDA) per-key-dimension decay across CPU, CUDA, and WebGPU, and extended `GatedDeltaNet` and `CausalConvWithState` state/gate handling for multi-token speculative decoding ([#33817](microsoft/onnxruntime#33817), [#33471](microsoft/onnxruntime#33471)). - Added an optional `query_seqlens` input to `GroupQueryAttention` to skip right-padded prefill tokens and idle batch entries (`query_seqlens[b] = 0`) on CPU and CUDA, and added `q_norm_epsilon` / `k_norm_epsilon` attributes to preserve distinct Q and K normalization epsilons during GQA pre-norm fusion ([#33503](microsoft/onnxruntime#33503), [#33505](microsoft/onnxruntime#33505), [#33832](microsoft/onnxruntime#33832), [#33363](microsoft/onnxruntime#33363)). - Defined the INT3 `QMoE` v1 packed-code contract, added opt-in post-SiLU clipping (`swiglu_limit_after_activation`) and tanh-`GEGLU` (`activation_type="geglu"`) support to MoE/QMoE schemas, and added `GatherPacked` (CPU/CUDA), `MatMulPacked` (CUDA, including native Bolt `global_scale`), and `MoEPacked` (CUDA) to consume native GGUF/block-packed quantized weights directly ([#33394](microsoft/onnxruntime#33394), [#33954](microsoft/onnxruntime#33954), [#33633](microsoft/onnxruntime#33633), [#33953](microsoft/onnxruntime#33953), [#33962](microsoft/onnxruntime#33962)). - Added `PackedSparseAttentionIndexerMerge` for multi-token prediction index sharing and optional `state_status` / mid-state outputs on `PackedSparseAttentionIndexer` across CUDA and WebGPU; extended `GatedAdd` with an optional projected `gate_weight` across CPU, CUDA, and WebGPU; and supported empty unpacked weights in CPU and WebGPU `MatMulNBits` ([#33224](microsoft/onnxruntime#33224), [#33383](microsoft/onnxruntime#33383), [#33472](microsoft/onnxruntime#33472), [#33470](microsoft/onnxruntime#33470), [#33559](microsoft/onnxruntime#33559)). - Registered core contrib schemas for provider-targeted LLM and speculative-decoding operators, including `SwiGLU`, `RMSNormRotary`, `HyperConnectionLowRankMix`, `RouterGateUpProj`, `SpeculativePrefixCommit`, `GatedDeltaNetStateReplay`, `ConvStateReplay`, and `MatMulNBitsQLoRA` ([#32126](microsoft/onnxruntime#32126), [#33469](microsoft/onnxruntime#33469), [#33384](microsoft/onnxruntime#33384), [#33650](microsoft/onnxruntime#33650), [#33474](microsoft/onnxruntime#33474), [#33213](microsoft/onnxruntime#33213), [#33493](microsoft/onnxruntime#33493), [#33370](microsoft/onnxruntime#33370)). ### Plugin Execution Provider Infrastructure - Fixed an `OrtMemoryInfo` ABI mismatch across the ORT/plugin EP boundary by converting `OrtMemoryInfo` instances exclusively through the C API ([#33571](microsoft/onnxruntime#33571)). - Zero-initialized `OrtEpFactory` structs in provider-bridge EP factories so unimplemented optional callbacks are null, and prevented double DLL initialization and handle leaks when probing and registering plugin EP libraries ([#33117](microsoft/onnxruntime#33117), [#33701](microsoft/onnxruntime#33701)). - Hardened portable model-package EP recreation, custom-domain preservation, UTF-8 path handling, and `shared_assets` path confinement and symlink validation ([#32790](microsoft/onnxruntime#32790), [#33826](microsoft/onnxruntime#33826)). ... (truncated) Commits viewable in [compare view](microsoft/onnxruntime@v1.31.1...v1.32.0). </details> Updated [Microsoft.NET.Test.Sdk](https://github.com/microsoft/vstest) from 18.11.0 to 18.12.0. <details> <summary>Release notes</summary> _Sourced from [Microsoft.NET.Test.Sdk's releases](https://github.com/microsoft/vstest/releases)._ ## 18.12.0 ## Security fix This release includes the fix for CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0 and earlier. Every version from 18.11.1 on is fixed, including 18.12.0. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Fix TestCaseStart/Stop events dropped for data-driven tests with shared TestCase.Id by @nohwnd in microsoft/vstest#16033 * Stop the discovery manager factory from shadowing discoveryCriteria by @nohwnd in microsoft/vstest#16391 * Gate the HTTP link checker on a fingerprint of the broken links by @nohwnd in microsoft/vstest#16413 * Quote boolean conditions in shipped Microsoft.TestPlatform.targets by @nohwnd in microsoft/vstest#16404 * Obsolete API hygiene: the compatibility-safe changes by @azat-msft in microsoft/vstest#16420 * Add xxHash128 test ids, available but not default by @azat-msft in microsoft/vstest#16378 * Isolate attachment manager test temporary files by @mmitche in microsoft/vstest#16463 * Test ids report logger by @azat-msft in microsoft/vstest#16443 * Isolate code coverage files in published output by @azat-msft in microsoft/vstest#16483 * Preserve OutputType for Android test projects by @jonathanpeppers in microsoft/vstest#16496 * Fix dotnet host resolution under proot (ARM64/Termux) by @azat-msft in microsoft/vstest#16462 * Mark the xxHash128 test id API as experimental by @azat-msft in microsoft/vstest#16465 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16633 * Fix localized resource paths in CLI source-build/non-Windows packages by @nohwnd in microsoft/vstest#16644 **Full Changelog**: microsoft/vstest@v18.11.0...v18.12.0 ## 18.11.1 ## Security fix This release fixes CVE-2026-105544, an elevation of privilege vulnerability in the VSTest artifact post-processor. The post-processor could consume a forged `executionComplete.json` written by another local user, and then run attacker-controlled output handling. Affected: `Microsoft.TestPlatform.CLI` 18.11.0. Fixed in: `Microsoft.TestPlatform.CLI` 18.11.1. `Microsoft.TestPlatform.CLI` is the VSTest build that we ship into the .NET SDK. The recommended way to get this fix is to update your .NET SDK to the latest patched version, see the advisory for the SDK versions. The fix is also released in the `Microsoft.TestPlatform.CLI` package, for completeness. That package is rarely used directly. It is not a dependency of `Microsoft.NET.Test.Sdk` or `Microsoft.TestPlatform`, so those packages do not bring it in. Details: microsoft/vstest#16654 and dotnet/announcements#517 Using custom Docker images or Nano Server? See microsoft/vstest#16652 ## What's Changed * Security fix for CVE-2026-105544, see microsoft/vstest#16654 * Ship Microsoft.Diagnostics.NETCore.Client in Microsoft.TestPlatform and Portable packages by @nohwnd in microsoft/vstest#16632 * Fix localized resource paths in CLI source-build packages by @nohwnd in microsoft/vstest#16643 **Full Changelog**: microsoft/vstest@v18.11.0...v18.11.1 Commits viewable in [compare view](microsoft/vstest@v18.11.0...v18.12.0). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from 4a95b8a to 5e6bea9 Compare October 19, 2026 09:00

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from 5e6bea9 to a0ce0a3 Compare October 20, 2026 09:22

Tianlei Wu (tianleiwu) changed the title ~~[CUDA] Add session-scoped QMoE offload lifecycle, device memory ledger, and arena shrink hooks~~ [CUDA] Add session-scoped MoE/QMoE offload planner, lifecycle controller, and memory budget ledger Oct 20, 2026

Tianlei Wu (tianleiwu) mentioned this pull request Oct 20, 2026

[CUDA] Add portable QMoE CPU/GPU offload backend with doorbell decode, adaptive promotion, and DMA prefill #33452

Merged

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from a0ce0a3 to f7e78c0 Compare October 20, 2026 10:09

Tianlei Wu (tianleiwu) added this pull request to stack #33495 October 20, 2026 18:52

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from f7e78c0 to 84ac121 Compare October 21, 2026 01:45

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from 84ac121 to e2289c7 Compare October 21, 2026 06:32

Tianlei Wu (tianleiwu) marked this pull request as ready for review October 21, 2026 18:13

Copilot AI balanced review requested due to automatic review settings October 21, 2026 18:13

Copilot started reviewing on behalf of Tianlei Wu (tianleiwu) October 21, 2026 18:14

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from e2289c7 to 0536a95 Compare October 22, 2026 01:25

Tianlei Wu (tianleiwu) requested review from Akshay Sonawane (apsonawane), kunal-vaishnavi and Ti-Tai Wang (titaiwangms) October 23, 2026 23:38

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from 0536a95 to 9eca322 Compare October 23, 2026 23:59

Tianlei Wu (tianleiwu) requested a balanced review from Copilot October 24, 2026 00:19

Copilot started reviewing on behalf of Tianlei Wu (tianleiwu) October 24, 2026 00:19

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Tianlei Wu (tianleiwu) force-pushed the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch from 9eca322 to 5d257d4 Compare October 24, 2026 03:59

Tianlei Wu (tianleiwu) mentioned this pull request Oct 24, 2026

[CUDA] Add INT4 and INT3 QMoE CPU/GPU offload support #33575

Merged

Tianlei Wu (tianleiwu) requested a balanced review from Copilot October 26, 2026 04:07

Copilot started reviewing on behalf of Tianlei Wu (tianleiwu) October 26, 2026 04:08

View session

Baiju Meswani (baijumeswani) previously approved these changes Oct 31, 2026

View reviewed changes

Tianlei Wu (tianleiwu) dismissed Baiju Meswani (baijumeswani)’s stale review via 0804bd5 October 31, 2026 01:12

Tianlei Wu (tianleiwu) enabled stacked auto-merge (squash) October 31, 2026 04:18

Baiju Meswani (baijumeswani) approved these changes Oct 31, 2026

View reviewed changes

Tianlei Wu (tianleiwu) merged commit 649f516 into main Oct 31, 2026

Tianlei Wu (tianleiwu) deleted the tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle branch October 31, 2026 04:27

Tianlei Wu (tianleiwu) mentioned this pull request Nov 3, 2026

Add CUDA weight loading with GPUDirect Storage and Microsoft DirectStorage #32713

Open

Sunghoon Choi (hanbitmyths) mentioned this pull request Nov 6, 2026

Harden CUDA graph ownership, retained-buffer lifetime, and failed-capture recovery #33932

Open

BrewTestBot mentioned this pull request Nov 13, 2026

onnxruntime 1.32.0 Homebrew/homebrew-core#325304

Merged

1 task done

This was referenced Nov 14, 2026

Bump the nuget-minor-and-patch group with 2 updates vitorafgomes/scout#14

Closed

Bump the nuget-minor-and-patch group with 2 updates siebe41/SemanticRouter.NET#387

Open

Bump coverlet.collector and 25 others Cezart2332/ridelance-backend#30

Open

Navigation Menu

Repository navigation

[CUDA] Add session-scoped MoE/QMoE offload planner, lifecycle controller, and memory budget ledger - #33451

[CUDA] Add session-scoped MoE/QMoE offload planner, lifecycle controller, and memory budget ledger#33451
Tianlei Wu (tianleiwu) merged 6 commits into
mainfrom
tlwu/20261019/qmoe-offload-p2-cuda-memory-lifecycle

Tianlei Wu (tianleiwu) commented Oct 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) commented Oct 31, 2026

Uh oh!

Tianlei Wu (tianleiwu) commented Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Tianlei Wu (tianleiwu) commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) commented Oct 31, 2026

Uh oh!

Tianlei Wu (tianleiwu) commented Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tianlei Wu (tianleiwu) commented Oct 19, 2026 •

edited

Loading