Repository navigation
Add a C API to serialize an OrtGraph to an ONNX model - #34014
Open
Danny On The Air (dannyota) wants to merge 2 commits into
Open
Danny On The Air (dannyota) wants to merge 2 commits into
Danny On The Air (dannyota) wants to merge 2 commits into
Conversation
Plugin EPs receive an OrtGraph but had no public way to turn it into ONNX ModelProto bytes, for example to hand a partition to a compiler. Add OrtGraphSerializationOptions with an external-initializer policy, a size threshold, an initializer-location callback and a write chunk size, plus Graph_SerializeToOnnxModel (allocator-owned bytes) and Graph_SerializeToOnnxModelWithWriteFunc (streaming). Only the supplied graph view is serialized, in Graph_GetNodes order; captured outer-scope values of a control-flow subgraph become inputs or copied initializers. Training info is not serialized. Output is deterministic for unchanged initializers and deterministic callbacks. C++ wrappers included. Graph keeps the original proto of sparse initializers whose values or indices use external data, so PRESERVE can keep those references after densification. This also fixes a dangling reference: Graph:: AddConstantProtoAsInitializer stored the name of a local TensorProto in sparse_tensor_names_, a set of string references. Signed-off-by: Danny <dannyontheair@outlook.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Serialization can dereference unvalidated model-supplied memory addresses in unused local functions, risking crashes or disclosure of process memory.
7 open issues
What changed in this PR
Adds public C and C++ APIs to serialize execution-provider graph views into ONNX model bytes, addressing #32799 without requiring plugin providers to implement their own conversion.
Changes:
- Adds allocator-owned and streaming output with configurable external-initializer policies.
- Serializes graph views, nested subgraphs, captured values, and local functions deterministically.
- Preserves sparse external-data references, fixes a sparse-name lifetime issue, and adds serialization tests.
| File | Description |
|---|---|
onnxruntime/test/ep_graph/test_ep_graph.cc |
Adds policy, ownership, streaming, sparse, and round-trip tests. |
onnxruntime/core/session/ort_apis.h |
Declares internal serialization API entry points. |
onnxruntime/core/session/onnxruntime_c_api.cc |
Appends serialization functions to the API table. |
onnxruntime/core/session/graph_serialization.cc |
Implements graph conversion, initializer policies, and output handling. |
onnxruntime/core/graph/model.h |
Declares graph-independent model metadata export. |
onnxruntime/core/graph/model.cc |
Copies model fields without graph or training information. |
onnxruntime/core/graph/graph.cc |
Preserves sparse references and fixes sparse-name ownership. |
include/onnxruntime/core/session/onnxruntime_cxx_inline.h |
Implements C++ serialization wrappers. |
include/onnxruntime/core/session/onnxruntime_cxx_api.h |
Declares C++ options and owned output types. |
include/onnxruntime/core/session/onnxruntime_c_api.h |
Defines and documents public serialization APIs. |
include/onnxruntime/core/graph/graph.h |
Adds sparse-source storage and serialization accessors. |
Review effort: Balanced
- Walk tensor attributes and local function bodies with one traversal, so FAIL_IF_PRESENT, EMBED and REPACKAGE cover them, and reject ORT in-memory address markers there before any callback. - Add initializers as graph inputs for IR version < 4. - Emit INT64 sparse indices. - Build opset imports from the view's live domain map. - Build externalized TensorProtos without copying the payload. - Guard the sparse test fixture with DISABLE_SPARSE_TENSORS. Signed-off-by: Danny <dannyontheair@outlook.com>
Chi Lo (chilo-ms)
pushed a commit
that referenced
this pull request
Nov 12, 2026
…4159) ### Description Forward the initializer-data callback into `OrtGraphToProto` subgraph recursion, keeping the same mutable callback state throughout a traversal. Preserve the empty-callback path and both public function signatures. The recursive call currently omits the callback, so nested initializers silently stay inline and nested callback errors are never reported. Passing the `std::function` by value instead would copy mutable captures into each branch, potentially reusing external-data offsets. A conditional `std::ref` wrapper shares the active callback during synchronous recursion without wrapping an empty function. Adds generated, CPU-only regressions in the existing EP graph suite: - root, sibling and depth-two initializers, including same-named initializers with different values; - a mutable value-captured offset, checking every external-data location/offset/length and the corresponding bytes; - external-file reload and actual inference for every `If` branch; - both GraphProto/ModelProto overloads, omitted/empty callbacks, and exact nested error propagation with no subsequent callbacks. #### Validation Built and tested on macOS arm64, Apple Clang 17, CPU-only Release shared-library build, based on `a8246c0d094f1ab0d6bf236228e5e67683321cd7`. No minimal build or operator exclusions were used. ```sh source .venv/bin/activate python tools/ci_build/build.py \ --build_dir build/MacOS --config Release --update --build --parallel 4 \ --target onnxruntime_ep_graph_test --build_shared_lib \ --cmake_generator Ninja --skip_submodule_sync --skip_tests \ --cmake_extra_defines FETCHCONTENT_TRY_FIND_PACKAGE_MODE=NEVER # Target-only builds do not run the general test target's fixture-copy step. cmake -E copy_directory onnxruntime/test/testdata build/MacOS/Release/testdata cd build/MacOS/Release ./onnxruntime_ep_graph_test --gtest_filter='EpGraphTest.*' ``` - **Before:** both added regressions fail against the unmodified header. The nested fixture serializes eight initializers but invokes the callback once; nested rejection incorrectly returns success. - **After:** all **20 EP graph tests pass**, including the two added regressions; no skips. - **Mutation checks:** forwarding the callback by value fails the offset/data/output assertions; unconditionally wrapping the empty callback fails the inline serialization control. The intended fix was restored, rebuilt and the full suite rerun afterward. - `lintrunner` and `git diff --check` pass. <details> <summary>Selected output from the actual local runs</summary> Baseline: ```text Expected equality of these values: callback_calls Which is: 1 inline_initializers.size() Which is: 8 Value of: status.IsOK() Actual: true Expected: false [ PASSED ] 0 tests. [ FAILED ] 2 tests ``` Restored final patch: ```text [==========] 20 tests from 1 test suite ran. (48 ms total) [ PASSED ] 20 tests. ``` </details> Windows execution has not been tested locally. Existing macOS telemetry duplicate-class and model-fixture warnings were not suppressed. ### Motivation and Context Fixes #34137. Thanks to @zadesai for the reproducer and the reference-forwarding approach. This fixes the existing header-only EP utility. #33921 concerns FLOAT16 tensor attributes, while #34014 adds a separate serialization C API; neither fixes this callback omission. AI assistance: OpenAI Codex assisted with implementation, independent source review, and execution of the reported local tests. The results above are from those actual automated runs. Signed-off-by: Silu Panda <31051721+SiluPanda@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements the API proposed in #32799: serialize an
OrtGraphto ONNXModelProtobytes.OrtGraphSerializationOptions(create/release) with setters for the external-initializer policy (PRESERVE, EMBED, REPACKAGE, FAIL), a minimum size threshold, anOrtGetInitializerLocationFunccallback and a maximum write chunk size.Graph_SerializeToOnnxModelreturns allocator-owned bytes;Graph_SerializeToOnnxModelWithWriteFuncstreams throughOrtWriteBufferFunc. Both reuse the existing callback types from the compile API.Graph_GetNodesorder, including nested subgraphs and local functions. Captured outer-scope values of a control-flow subgraph become graph inputs or copied initializers. Training info is not serialized.Graph's checks.Ort::GraphSerializationOptionsand twoConstGraphmethods.Graphnow keeps the original proto of sparse initializers whose values or indices use external data, so PRESERVE keeps those references after densification. This also fixes a dangling reference on main:Graph::AddConstantProtoAsInitializerinserted the name of a localTensorProtointosparse_tensor_names_, a set ofstd::reference_wrapper<const std::string>.Testing (macOS arm64 and Linux x64,
--build_shared_lib, same results on both)onnxruntime_ep_graph_test --gtest_filter='EpGraphTest.*': 32 pass, including 14 newSerializeOnnx_*tests: streaming vs allocated bytes, chunking and first-error stop, failure outputs and allocator ownership, all initializer policies and the threshold, filtered views and nested/captured values, local functions and their tensor attributes, rejected in-memory markers, live opset imports, IR 3 and sparse outputs checked with the ONNX checker, sparse Constants,Ifinlining, training info, determinism, and a REPACKAGE round trip that reloads the model and compares CPU outputs.onnxruntime_test_allGraph, sparse, inlining,Ifand Constant tests: 229 pass.onnxruntime_DISABLE_SPARSE_TENSORS=ON:EpGraphTest.*28 pass; the 4 sparse-only tests are compiled out.Motivation and Context
Fixes #32799. Plugin EPs receive an
OrtGraphbut had no public way to get ONNX bytes for it, for example to hand a partition to a compiler.