Repository navigation
fix(windows): support long paths in scan tooling - #435
faizan-oai wants to merge 43 commits into
Conversation
…-long-path-inventories
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
codex-security/sdk/typescript/_bundled_plugin/scripts/workbench_db.py
Lines 3494 to 3495 in 054f72a
On Windows, when an artifact path reaches the 248-character threshold, filesystem_path(candidate) adds the \\?\ prefix, while the persisted candidate passed by scan_result remains portable. This raw comparison therefore rejects the valid artifact (and candidate.is_file() may also fail on its unprefixed long spelling), causing completed scans to omit reports, findings, and SARIF from their returned artifact map. Normalize both paths with portable_path and inspect the resolved filesystem path instead.
AGENTS.md reference: sdk/typescript/AGENTS.md:L23-L24
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4d2f045f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6dc0fd8cca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9a7f97c7f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Security review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fa7731e22
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 732a69abf2
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
|
@codex review Please check that long Windows paths and rename-source fallback both work in the updated inventory code. Updated head: |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
mldangelo-oai
left a comment
There was a problem hiding this comment.
Please hold this branch while reconciling it with current main. I confirmed two path regressions at af727e0.
database_path()prefixes the filename before SQLite can select its spelling. With a 1,030-byte ordinary filename, the receipt command handed SQLite 1,034 bytes for a drive path and 1,036 for UNC. Both exited 73 before the file-open callback. Main returned the seeded receipt for both. This used the real SQLite pager on Linux with the default Win32 pathname budget simulated. Preserve the ordinary spelling through filename selection, including for writable opens, and carry the byte-boundary coverage from #2386.- The archive relocation callback joins the relative suffix to an ordinary archive root. For a stored Windows worker result with a verbatim-only leaf such as
result.json., the rebased string loses the namespace that identifies that file. This was reproduced lexically with the branch's callback. Rebase under the extended archive root, then convert the complete result to ordinary spelling only if it still denotes the same object. Keep a regression for an ordinary sibling with the trailing dot removed.
A merge with current main conflicts in 13 files, including the target registration and inventory code. Preserve main’s checkout-generation ownership checks and Windows output-directory protections when porting the changes. The new rejection of existing verbatim-only target directories also needs an explicit compatibility decision or a narrower fix. The earlier green CI predates this integration.
The branch-head dependency audit flags the older optional dependency. Keep the existing Promptfoo features and resolve the fixed release within its declared range.
|
#2689 ports the writable workbench database and scan archive cases from this PR onto current main. It covers drive and UNC filename selection for SQLite, matching an existing scan across ordinary and extended path spellings, and preserving a verbatim trailing-dot file during archival. It keeps the checkout and workbench state protections on main. The broader inventory and target-registration changes remain here. #2386 covers read-only finding receipt inspection independently. All seven new native cases passed on Windows Server 2025, including CLI writes and reads at a 1,030-byte drive filename through both input spellings. The complete Source CI run passed. UNC paths and legacy policy were exercised in simulation. Windows JUnit report |
Summary
Windows scans failed when repository paths, scan state, generated artifact names, or Deep Scan directories exceeded legacy Win32 path limits. Filesystem operations now use extended-length Windows paths while persisted state and public scan results retain ordinary portable paths.
Integration with current main is on hold. The original merge trial against
173483d7produced 13 conflicts, including target registration and workbench storage. Atdd156058there are 14. Main has already removed the optional dependency chain that the branch updates in its eval lockfile.Changes
Add shared, UTF-16-aware helpers for Windows filesystem paths and portable path spelling.
Support long paths throughout inventory generation, candidate normalization and ranking, project configuration, security-policy resolution, scan finalization, exports, and workbench state.
Preserve historical scans and target identities across ordinary and legacy extended path spellings without rewriting sealed scan history.
Normalize complete Deep Scan artifact, heartbeat, ledger, publication, backup, and generated scan-directory paths.
Integrate the then-current
mainand its canonical plugin source layout, publication module, case-insensitive Windows history queries, scope handling, and publication-copy recovery.Compare portable parent paths when archiving a scan whose longer archive name crosses the Windows path threshold. Add native Windows coverage for that boundary and for legacy scan-root aliases.
Derive the Windows long-filename fixture from its actual temporary directory, so it still exceeds 260 UTF-16 code units under short CI roots.
Exercise real Windows long-path behavior using bundled ripgrep, and exclude generated Python bytecode from published packages. Keep the scope parser test isolated from the real filesystem when the shared helper constructs paths.
Preserve the ordinary SQLite filename when an otherwise equivalent drive or UNC prefix alone exceeds the default pager budget. Filesystem metadata still uses the extended path.
Preserve the identity of a verbatim-only child during archive relocation and recovery, including when a distinct ordinary sibling exists.
Add a native Windows workbench write and read at the SQLite filename limit. Update the old eval lockfile within its existing dependency ranges and carry over the Ubuntu profile setup from #2418 for branch CI.
Testing
Earlier branch revisions:
12345) and randomized SDK suites at merge commit9c699ae: 2,111 passed, 44 skipped, zero failures each.git diff --checkpassed.On the first repair head, the path and archive suites passed on Linux (73 passed, four Windows-only skips). A real SQLite pager on Linux with simulated Win32 path handling opened the 1,030-byte drive and UNC filenames read-only and writable. The original branch exited before the OS open. The first full branch run also identified an older eval lockfile audit failure and Ubuntu 26.04 AppArmor profile lookup failure.
On
dd156058, the native Windows Server 2025 / Python 3.12 job passed 482 tests and 40 subtests with zero skips, failures or errors. The JUnit report records a successful workbench CLI write and read at a 1,030-byte ordinary drive filename and recovery ofcoverage.json.after archiving withcoverage.jsonpresent. UNC and enforcement of legacy Windows policy were simulated separately.The dependency update preserves the optional eval features. A frozen install and the complete deterministic eval suite passed. With the same Trivy 0.76.1 database, the 724-component eval inventory went from two HIGH findings to zero HIGH or CRITICAL findings. Bash syntax and both AppArmor profile layouts, missing profiles, and preference when both exist were checked. Ruff lint and format, the SDK CI build, plugin source compatibility and its nine Node tests passed. The full CI dispatch passed all 65 jobs on this branch head, including the Ubuntu installed-package job. The PR conflicts with main, so the integrated result still needs CI.
Risk and rollout
Extended-length spelling is limited to Windows filesystem access. Persisted paths, scan metadata, target identities, completed artifacts, containment checks, and non-Windows behavior need revalidation against current main. The existing test rejects verbatim-only target directories. Main accepts those directories at target validation and independently checks repository generation before reusing a target ID. Preserve generation ownership and decide the intended directory compatibility before landing. The receipt-specific fix is reviewed separately in #2386. The hosted Windows job covers a local drive. A native UNC share and a Windows host enforcing legacy path policy remain unverified.
Public disclosure review
The checklist below was completed for earlier work. The incremental repair review covered the branch, title, updated description, new commits, diff, synthetic test data and public references.