Repository navigation
Add stable EPContext data callbacks to OpenVINO - #32267
Open
Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 22 commits into
Open
Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 22 commits into
Gopalakrishnan Nallasamy (GopalakrishnanN) wants to merge 22 commits into
Conversation
Promote named-buffer EPContext callbacks to the stable C and C++ APIs with a required finite read-size policy. Add immutable EP config snapshots, strict participating-EP support negotiation, V28 compatibility shims, hardened sample helpers, example plugin adoption, documentation, and focused API/auto-EP tests.
Mark the strict-read flag unused when ONNX partitioning is excluded, and preserve the V28 EPContext config shims directly in minimal builds where the plugin EP API implementation is intentionally not linked.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Route native external OpenVINO compiled blobs through stable application callbacks, reject incompatible shared-context and OVIR read modes, deserialize each distinct callback name once, enforce finite non-empty payloads, and add provider-backed round-trip, error, size, deduplication, and no-sidecar coverage.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Gopalakrishnan Nallasamy (GopalakrishnanN)
force-pushed
the
GopalakrishnanN/openvino-epcontext-data-io
branch
from
August 31, 2026 08:24
d3cfa4b to
7805d4d
Compare
Use opaque read options consistently, clarify application/EP size enforcement, update version-31 callback support negotiation, and cover ownership and older-plugin boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…untime/contrib_ops/cuda/moe/moe_quantization.cc. (#32932) ### Description <!-- Describe your changes. --> Replace old use of `routing_record` with `routing_snapshot_` in onnxruntime/contrib_ops/cuda/moe/moe_quantization.cc. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Fix build error about undeclared identifier `routing_record`. Likely an unlucky merge artifact from two recent PRs #32761 and #32738. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Increase the minimal Android section-size budget by 2 KiB. CI measured 1,598,685 bytes versus the prior 1,597,440-byte budget; the new 1,599,488-byte limit retains 803 bytes of headroom. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement review option 3: remove the proposed read-options APIs and runtime size-limit storage, keep callback/state snapshots and strict support negotiation, and leave allocation/deserialization limits to applications and EPs. Retain an EP-selected bound in the sample helper with boundary coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Clarify the external-data lifecycle query, simplify callback flag documentation, keep allocation-policy guidance in the design document, and require explicit C++ callback state arguments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use callback/state-only transport and renamed callback support flags. Move test allocation limits into the application callback, preserve filesystem-path regression coverage and error guidance, and validate the OpenVINO 2026.2.1 CPU path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Updated to the finalized API in #32265 via f28a3c4. The provider and both factories now use callback/state-only transport and the renamed support query/flags. Allocation-limit tests enforce policy inside the application callback before allocating; provider payload validation and no-fallback behavior remain intact. The merge preserves both callback loading and the newer filesystem-path tests, including their actionable configuration guidance. Built with the CI OpenVINO 2026.2.1 SDK and ran all 13 focused EPContext tests on Intel CPU: all passed, no skips. Formatting and diff checks passed. The PR remains stacked on #32265; GPU/NPU and generic-device API runtime tests were not run locally. |
Tianlei Wu (tianleiwu)
left a comment
Tianlei Wu (tianleiwu)
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the complete isolated OpenVINO diff at f28a3c4 against stacked base 46f94aa, including both factory paths, callback stream ownership/bounded seeking, native-blob deserialization, write error propagation, sharing restrictions, OVIR path changes, and affected tests. No actionable code findings.
Callback buffers are guarded on failure paths and deserialized into owned native-blob storage before stream destruction. Native callback writes avoid sidecars, unsupported OVIR reads reject callbacks rather than falling back to disk, and the tests retain independent filesystem-path coverage alongside callback-only loading.
CI: Optional Lint C++, Optional Lint, lintrunner, Python format, and license/cla passed; Linux Android Emulator QNN CI Pipeline is NEUTRAL. No failed or pending checks. There is no OpenVINO build/runtime check in the current rollup, so the author-reported CPU results are not independent CI validation; GPU/NPU and generic-device runtime coverage remain unvalidated.
Merge status: the declared stacked base is MERGEABLE/CLEAN, and merge-tree against the latest parent branch 1cbc5d1 is clean. Against current origin/main 0ea42b2, inherited conflicts remain in .github/workflows/android.yml and onnxruntime/test/framework/moe_expert_counting_test.cc; these are outside the isolated OpenVINO changes and are not conflicts with the declared base. Please account for them when integrating/retargeting the stack.
Static review only; no local builds, tests, linters, or runtime verification were run.
Gopalakrishnan Nallasamy (GopalakrishnanN)
added a commit
that referenced
this pull request
Oct 5, 2026
### Description <!-- Describe your changes. --> Promotes the experimental EPContext named-buffer callback transport to the stable ONNX Runtime 1.31 C and C++ APIs. - Adds stable read/write callback registration and immutable EP-facing snapshots of the callback and application-state pointers. - Implements review option 3: removes `max_data_size` and the proposed read-options type, constructor, getter, setter, and release APIs. Registration and retrieval carry only callback/state pointers. - Leaves allocation limits to the application callback and data validation/deserialization limits to the EP. ORT does not invoke the callback or inspect its result. The sample helper can enforce an EP-selected limit without putting that policy into the stable ABI. - Adds append-only `GetEpContextDataCallbackSupport` READ/WRITE advertisement, gated on plugin API version 31. Unsupported participating EPs are rejected before `Compile()`, including the ORT-format read path. Registered callbacks cannot silently fall back to filesystem I/O. - Removes the experimental EPContext lookup entries and C++ wrapper; experimental APIs have no stability guarantee. - Updates the C++ wrappers, provider bridge, example EP, tests, and design document. - Retains the early support query for providers that can produce external context data without `Compile()`, with VitisAI's direct-assignment lifecycle rationale documented. - Includes the upstream GPU build fix from #32932 and the previously approved 2 KiB Android size-budget adjustment. Provider/binding adoption follows in QNN (#32266), OpenVINO (#32267), C# (#32270), and Python (#32271). Those follow-ups need to use the finalized callback/state-only signatures. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> This graduates the transport introduced experimentally by #28624 and refined by #29294 and #31195. Applications can keep external EPContext data in encrypted or application-managed storage without silently falling back to plaintext filesystem I/O. Following the payload-policy review, the stable contract is limited to transport, ownership, and support negotiation. Centralized result-size validation would not itself prevent allocation inside the callback; passing a limit to the callback would still depend on callback/EP enforcement. Removing the policy avoids permanent API surface for a guarantee ORT does not provide. ### Testing Windows VS18 builds: - `cmake --build build\WindowsEpContextStable\RelWithDebInfo --config RelWithDebInfo --parallel 8 --target onnxruntime_shared_lib_test onnxruntime_autoep_test onnxruntime_test_all` — passed. - `cmake --build build\WindowsEpContextMinimal\MinSizeRel --config MinSizeRel --parallel 4 --target onnxruntime_shared_lib_test` — passed; minimal configuration disables exceptions, RTTI, and ML ops. Tests run from `build\WindowsEpContextStable\RelWithDebInfo\RelWithDebInfo`: - `.\onnxruntime_shared_lib_test.exe --gtest_filter=EpContextDataApiTest.*` — 14 passed, including immutable snapshots, cloning/lifetime, unchanged error outputs, and callback clearing. - `.\onnxruntime_autoep_test.exe --gtest_filter=*EpContext*` — 25 passed, 2 skipped for unavailable Windows symlink privileges. Covers strict support negotiation, callback/file paths, and EP-selected helper size-limit boundaries. - `.\onnxruntime_test_all.exe --gtest_filter=InternalTestingEP.*EpContext*:InternalTestingEP.Nhwc*` — 19 passed. Tests run from `build\WindowsEpContextMinimal\MinSizeRel\MinSizeRel`: - `.\onnxruntime_shared_lib_test.exe --gtest_filter=EpContextDataApiTest.*` — 2 passed. Formatting and `git diff --check` passed. No references to the removed read-options API or runtime size fields remain in tracked files. Not validated locally: GPU/Android builds, VitisAI/VAIP hardware behavior, other hardware EP integrations, or the full suite. The VitisAI lifecycle discussion remains open for reviewer confirmation. --------- Co-authored-by: Gopalakrishnan Nallasamy <gopalakrishnan.nallasamy@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Base automatically changed from
GopalakrishnanN/stable-epcontext-data-api
to
main
October 5, 2026 21:31
Retain the landed API and partitioner foundation. Preserve native callback import/export, document OVIR and sharing exclusions, reject callback-backed external shared weights, and add SDK-independent ownership and seek-boundary tests for the production stream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Gopalakrishnan Nallasamy (GopalakrishnanN)
October 7, 2026 19:56
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The callback helper has a compilation issue and malformed payloads can trigger unbounded deserializer allocations.
3 open findings
What changed in this PR
Adds stable EPContext named-buffer callback support to OpenVINO for application-managed compiled blobs.
Changes:
- Routes native EPContext imports and exports through read/write callbacks.
- Adds callback-backed stream ownership, validation, and provider tests.
- Preserves filesystem OVIR handling and documents callback limitations.
| File | Description |
|---|---|
onnxruntime/core/providers/openvino/backends/basic_backend.cc |
Uses resolved OVIR model paths. |
onnxruntime/core/providers/openvino/backend_manager.cc |
Selects callback-backed model streams. |
onnxruntime/core/providers/openvino/callback_buffer_stream.h |
Adds an owning seekable callback-buffer stream. |
onnxruntime/core/providers/openvino/contexts.h |
Stores callback functions and states. |
onnxruntime/core/providers/openvino/onnx_ctx_model_helper.cc |
Reads, validates, and deserializes callback payloads. |
onnxruntime/core/providers/openvino/onnx_ctx_model_helper.h |
Extends model-stream metadata and APIs. |
onnxruntime/core/providers/openvino/openvino_execution_provider.cc |
Writes external context data through callbacks. |
onnxruntime/core/providers/openvino/openvino_execution_provider.h |
Advertises callback support. |
onnxruntime/core/providers/openvino/openvino_provider_factory.cc |
Copies callbacks through both factory paths. |
onnxruntime/core/providers/openvino/ov_interface.cc |
Accepts the resolved OVIR XML path. |
onnxruntime/core/providers/openvino/ov_interface.h |
Renames the OVIR path parameter. |
onnxruntime/test/providers/openvino/callback_buffer_stream_test.cc |
Tests stream seeking and allocation release. |
onnxruntime/test/providers/openvino/openvino_ep_context_test.cc |
Tests callback import/export and rejection paths. |
docs/design/Compiled_Model_Encryption.md |
Documents OpenVINO callback behavior. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Gopalakrishnan Nallasamy (GopalakrishnanN)
requested a review
from Tianlei Wu (tianleiwu)
October 7, 2026 21:20
…o OpenVINO adoption Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…deduplication Qualify anonymous-namespace Status helpers; reject out-of-stream and negative native blob metadata before allocating. Exercise repeated and distinct external names in one session and malformed callback payloads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
approved these changes
Oct 21, 2026
Tianlei Wu (tianleiwu)
left a comment
Tianlei Wu (tianleiwu)
left a comment
Contributor
There was a problem hiding this comment.
LGTM — thanks for rebasing onto main, adding stream-bounds and cumulative-size validation in BinManager::DeserializeImpl, and expanding the single-session deduplication and malformed-payload tests.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds OpenVINO provider adoption of stable named-buffer EPContext callbacks already merged in #32265.
397657de778d05b17c0249ba37c0b4290e1ab3b5. The final delta is 14 provider/test/documentation paths, not the original 54-file foundation stack.Preserves main's API ordering, support negotiation, filesystem path containment, and unrelated providers. No public API version bump.
Motivation and Context
Applications need native OpenVINO context payloads in managed/encrypted storage with explicit error/ownership behavior. The foundation is already merged, so this is a main-based provider delta rather than a PR waiting for #32265.
Testing
Actual OpenVINO SDK 2026.2.1.21919 was found in the existing cache and reused read-only. Fresh isolated Windows CPU build after
setupvars.batandvcvars64.bat:Build passed, exit 0. Unsuppressed latest main independently reproduced
C2220/C4702in Abseilraw_hash_set.h:3245-3247while compilingfunction_utils.cc. Final validation removed global_CL_=/wd4702and instead used a local vendor-header-only pragma aroundfind_or_prepare_insert_small. No repository warning policy changed. This vendor-qualified build is not unrestricted/WXcoverage; the original diagnostic and exact local patch are retained.From the fresh executable directory:
Final result: 16/16 passed, no skips/failures, process exit 0, no leaks detected.
The initial unknown-symbol leak reports were reproduced by matching main zero-test controls without linker PDBs. Rebuilding with
/DEBUGPDBs lets the existing symbol-name allowlist identify intentional dependency allocations; prepared tests then exit cleanly. The checker remained enabled and executed, and its allowlist was unchanged.Main's symbolized legacy five-case control had no leaks but one existing path-guidance assertion failure (
ExportEpCtxFromOVIRModel/embed_off). The prepared implementation already provides the required guidance and passes the corresponding case. This is distinct from the corrected PDB/checker issue.Separately, SDK-independent compilation of the actual production callback stream passed and its 3 ownership/read/seek tests passed with clean exit 0.
The subsequent latest-main merge changes only CUDA's unrelated
is_int8predicate; it does not change this SDK-tested provider delta.Historical earlier-head evidence: the 13-case OpenVINO callback/EPContext/OVIR selector passed. The new results above supersede that claim for current preparation and include the additional production stream cases.
Still required: real GPU/NPU execution, current-head CI, and human approval. The vendor warning qualification is explicit. Main's independent materialized dtype CI blocker is tracked in #33183; failures are not waived.