Repository navigation
Enable Robust Symlink Support for External Data (HF Cache Support) - #27374
Merged
Merged
Conversation
Tianlei Wu (tianleiwu)
requested review from
Copilot and
Dmitri Smirnov (yuslepukhin)
February 18, 2026 05:21
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Tianlei Wu (tianleiwu)
requested a review
from Hariharan Seshadri (hariharans29)
February 18, 2026 05:23
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates ONNX Runtime’s external data path validation to properly handle models stored in symlinked directory layouts (notably Hugging Face Hub cache layouts), while keeping path-escape protections in place.
Changes:
- Extend
utils::ValidateExternalDataPathto optionally validate against the canonical (real) model directory when the logicalbase_dircheck fails. - Update
Graph::ConvertInitializersIntoOrtValuesto passmodel_pathintoValidateExternalDataPath. - Remove provider-bridge exposure of
ValidateExternalDataPathand add a Python symlink-based regression test.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| onnxruntime/core/framework/tensorprotoutils.h | Extends ValidateExternalDataPath API to accept optional model_path for canonical validation. |
| onnxruntime/core/framework/tensorprotoutils.cc | Implements dual logical+canonical directory validation for external data paths. |
| onnxruntime/core/graph/graph.cc | Passes model_path into external data validation for initializer loading. |
| onnxruntime/core/session/provider_bridge_ort.cc | Removes provider-host bridge method for external data path validation. |
| onnxruntime/core/providers/shared_library/provider_interfaces.h | Removes Utils__ValidateExternalDataPath from the ProviderHost interface. |
| onnxruntime/core/providers/shared_library/provider_api.h | Removes inline ValidateExternalDataPath wrapper from provider API. |
| onnxruntime/test/python/onnxruntime_test_python_symlink_data.py | Adds Python unittest coverage for HF-style symlink layouts and a negative escape case. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Edward Chen (edgchen1)
approved these changes
Feb 18, 2026
Tianlei Wu (tianleiwu)
added a commit
that referenced
this pull request
Feb 18, 2026
…27374) ## Summary This PR enables ONNX Runtime to correctly load models with external data when they are stored in symlinked directory structures. This is a common scenario for models cached by the Hugging Face Hub, where both the model file and data files are symlinks pointing into a flat `blobs/` directory. ## The Problem Previously, ONNX Runtime's external data path validation would only check if the resolved data path was under the logical directory of the model. In symlinked structures (like Hugging Face's `snapshots/` and `blobs/` layout), the resolved data path often sits in a different physical directory than the logical model path, leading to a "path escapes model directory" error even when the data is safely associated with the model. ## The Fix I have updated `ValidateExternalDataPath` to implement a dual-check mechanism: 1. **Logical Check:** Verify if the resolved data path is under the provided `base_dir` (the directory where the model was loaded from). 2. **Physical Check:** If the logical check fails, verify if the resolved data path is under the parent directory of the **real/canonical** model path. This approach ensures that models can load external data from both their logical siblings and their physical siblings while maintaining security constraints. ## Changes - **Core Logic:** - `onnxruntime/core/framework/tensorprotoutils.cc/h`: Updated `ValidateExternalDataPath` to accept `model_path` and perform the dual-check. - `onnxruntime/core/graph/graph.cc`: Updated call site in `ConvertInitializersIntoOrtValues` to pass the `model_path`. - **Cleanup:** - Removed unused `ValidateExternalDataPath` from `provider_api.h`, `provider_interfaces.h`, and `provider_bridge_ort.cc`. These were redundant as path validation is handled by core during session initialization. Provider has no need to validate it. - **Testing:** - Added `onnxruntime/test/python/onnxruntime_test_python_symlink_data.py` to simulate the Hugging Face Hub symlink structure and verify the fix.
Tianlei Wu (tianleiwu)
added a commit
that referenced
this pull request
Feb 18, 2026
This cherry-picks the following commits for the release: - #27350 [Build] Fix python packaging pipeline - #27349 [Build] Fix DML Nuget Pipeline for Release - #27334 [DNNL] Fix DNNL build - #27374 Enable Robust Symlink Support for External Data (HF Cache Support) --------- Co-authored-by: Hariharan Seshadri <shariharan91@gmail.com>
This was referenced Feb 23, 2026
This was referenced Jul 1, 2026
This was referenced Jul 8, 2026
This was referenced Jul 22, 2026
This was referenced Aug 1, 2026
This was referenced Aug 8, 2026
This was referenced Aug 17, 2026
This was referenced Aug 27, 2026
This was referenced Sep 17, 2026
This was referenced Oct 8, 2026
2 of 3 tasks
Tianlei Wu (tianleiwu)
added a commit
that referenced
this pull request
Nov 13, 2026
…34099) ### Description Since `huggingface_hub` 1.32, the Hugging Face Hub cache stores Xet files once per cache in a [blob store shared across repos](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#shared-blobs-across-repos) (`<cache>/blobs/<hash prefix>/<hash>`), and the repo's `blobs/<etag>` becomes a relative symlink to it. A model and its external data in one snapshot can then resolve into different folders (for example `blobs/28/…` and `blobs/d4/…`, or one in the repo's `blobs/` and one in the shared store). The symlink fallback from #27374 only accepts external data next to the resolved model, so models with external data loaded from a default `hf_hub_download` path fail with `External data path escapes model directory`. This PR teaches both external data validators about the shared store without trusting anything outside the cache's blob folders. ### Summary of Changes #### Validation (`onnxruntime/core/framework/tensorprotoutils.cc`) New Pass 3 in `ValidateExternalDataPath`, reached only when the model is a symlink and Passes 1–2 fail. If the canonical model directory is in a snapshot of a cache repo (`<cache>/<type>s--<repo id>/snapshots/<revision>/…`, also from a subfolder such as `onnx/`), external data is accepted when: 1. its path does not leave the model directory before symlinks are resolved (no `..` escape), and 2. both the model and the external data resolve to files under the blob stores of that cache: the repo's `blobs/` folder or the cache-wide `blobs/` folder. `ValidateOpenedExternalDataPath` applies the same blob-store check to the canonical path of the opened file, so the opened-file validation from #33063 also covers the new case. The cost is one extra canonicalization, and only after the existing passes fail. | File | Change | |------|--------| | `onnxruntime/core/framework/tensorprotoutils.cc` | Add `IsInHuggingFaceCacheBlobStores` and use it in both validators. | | `onnxruntime/test/framework/tensorutils_test.cc` | `HuggingFaceCachePathValidationTest`: 11 tests on a cache built like `huggingface_hub` does, with two-hop relative symlinks (6 accepted layouts, 5 rejected). | | `onnxruntime/test/python/onnxruntime_test_python_symlink_data.py` | End-to-end `InferenceSession` test for the shared store and for a mixed repo-local/shared layout. | #### Security notes - Paths are compared in canonical form. A canonical path has no symlinks, so a file whose canonical path is under `<cache>/blobs` is stored in that real folder. If a `blobs/` folder is itself a symlink or junction, nothing matches (`huggingface_hub` also never uses a symlinked `blobs/` folder as the shared store). - The model itself must resolve into the blob stores. This rejects symlinked models that point outside the cache, and `symlink/..` tricks in the model path. - The no-`..` requirement keeps a malicious model from reaching other cached repos' files. In a real cache, the entries of a snapshot folder are symlinks that `huggingface_hub` creates to files of the same repo. - Someone who can write into the cache root gains nothing new. With the existing Pass 2 (for example ORT 1.31.1), they can already make a model read any blob by putting the model file in the same shard folder (verified). `huggingface_hub` also gives the store and shard folders the same permissions as the cache root. - Unchanged: empty, absolute, and in-memory-tag locations are still rejected before any fallback, and Passes 1–2 behave as before. #### Comparison with #33974 #33974 targets the same issue. It also checks the `.huggingface-shared-blobs` marker (version `1`) and blob names (hex, 40/64 characters, 2-character shard), and requires a `models--` repo folder. These checks are not security boundaries, since anyone who can create the layout can also create the marker and names. They do tie ORT to `huggingface_hub` internals, so a future layout change would break loading again. I built both and loaded the same synthetic caches: | Scenario | #33974 | This PR | |---|---|---| | Shared store, model and data in different shards | loads | loads | | Mixed repo-local/shared blobs (both directions) | loads | loads | | `datasets--…` repo | fails | loads | | Model folder reached through a symlinked directory | fails | loads | | Shared store marker version other than `1` | fails | loads | | Location `../x`; data symlink outside the cache; symlinked `blobs/` | rejected | rejected | ### Motivation and Context Fixes #33950. Downstream projects currently work around it by pinning `huggingface_hub<=1.31`, setting `HF_HUB_DISABLE_SHARED_BLOBS=1`, or copying files out of the cache. ### Testing - Real cache: reproduced the issue with `onnxruntime==1.31.1` and `huggingface_hub==2.4.0` (`onnx-community/sam2.1-hiera-tiny-ONNX`, fresh `HF_HOME`). With this PR, the same cache loads. The two mixed layouts created by downloading one of the files with `HF_HUB_DISABLE_SHARED_BLOBS=1` also fail with 1.31.1 and load with this PR. - `onnxruntime_test_all --gtest_filter='HuggingFaceCachePathValidationTest.*:PathValidationTest.*:GetExtDataFromTensorProtoTest.*'`: all pass. Without the `tensorprotoutils.cc` change, the 6 tests for accepted layouts fail. Removing any one of the conditions (no `..` escape, model in a blob store, `blobs/` folder bound, repo folder name, symlinked `blobs/`, opened-file check) makes the targeted test fail. - `python -m pytest onnxruntime/test/python/onnxruntime_test_python_symlink_data.py`: 4 passed (plus 2 subtests). - Full `onnxruntime_test_all` on Linux x64 (CPU, Release): 2897 passed. `lintrunner`: no issues. - Not run locally: Windows and WASM. The new tests skip when symlinks cannot be created, and the load test is excluded on WASM like the other external-file tests in that file. ### Checklist - [x] Tests added/updated - [x] No breaking changes - [ ] Documentation updated (not applicable) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR enables ONNX Runtime to correctly load models with external data when they are stored in symlinked directory structures. This is a common scenario for models cached by the Hugging Face Hub, where both the model file and data files are symlinks pointing into a flat
blobs/directory.The Problem
Previously, ONNX Runtime's external data path validation would only check if the resolved data path was under the logical directory of the model. In symlinked structures (like Hugging Face's
snapshots/andblobs/layout), the resolved data path often sits in a different physical directory than the logical model path, leading to a "path escapes model directory" error even when the data is safely associated with the model.The Fix
I have updated
ValidateExternalDataPathto implement a dual-check mechanism:base_dir(the directory where the model was loaded from).This approach ensures that models can load external data from both their logical siblings and their physical siblings while maintaining security constraints.
Changes
onnxruntime/core/framework/tensorprotoutils.cc/h: UpdatedValidateExternalDataPathto acceptmodel_pathand perform the dual-check.onnxruntime/core/graph/graph.cc: Updated call site inConvertInitializersIntoOrtValuesto pass themodel_path.ValidateExternalDataPathfromprovider_api.h,provider_interfaces.h, andprovider_bridge_ort.cc. These were redundant as path validation is handled by core during session initialization. Provider has no need to validate it.onnxruntime/test/python/onnxruntime_test_python_symlink_data.pyto simulate the Hugging Face Hub symlink structure and verify the fix.