Skip to content

feat(python): follow sys.modules shims to the module they install - #1133

Merged
4 commits merged into
tirth8205:stagingfrom
JacksonFuck:feature/python-module-alias-shims
Nov 6, 2026
Merged

4 commits merged into
tirth8205:stagingfrom
JacksonFuck:feature/python-module-alias-shims

Conversation

@JacksonFuck

@JacksonFuck JacksonFuck commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request

Linked issue

No linked issue. This came up indexing a real backend that keeps ~70 flattened-import compatibility shims at its root. The behaviour and the fix are described in full below.

What & why

Some Python codebases keep old flat import paths working with a shim module that swaps itself out of sys.modules:

# billing.py
import importlib
import sys

sys.modules[__name__] = importlib.import_module("app.services.billing")

After import billing, the caller holds app/services/billing.py. The parser stopped at the shim file:

Call site Before After
import billing (IMPORTS_FROM) billing.py app/services/billing.py
billing.charge(1) bare charge, unresolved (charge is not top-level in the shim) app/services/billing.py::charge
billing.run_model(1) (re-exported by the real module) bare, unresolved app/services/engine.py::run_model
from billing import charge billing.py::charge (no such node) app/services/billing.py::charge
from billing import charge as pay; pay() billing.py::pay app/services/billing.py::charge
TESTED_BY from a test calling billing.charge bare source charge app/services/billing.py::charge
billing.refund(1) (not defined anywhere) bare bare (unchanged, as it should be)

On the codebase that prompted this, a module with 39 importers showed 1 (the shim), and a function with 59 callers showed 3.

Changes

  1. parser.py: follow module-alias shims during resolution. _python_module_alias_target() maps a resolved Python module file to the module it installs as sys.modules[__name__]. It is applied wherever Python imports become files: _resolve_module_to_file, _python_import_origin (so star exports and re-export origins see through shims too), and _python_submodule_file. Detection uses ast and accepts only the literal, module-level form, importlib.import_module("dotted.name") or a bare import_module(...). A computed or relative name, a swap inside a block, or a target outside the repository leaves the file as the module. Chains of shims are followed up to 5 hops, with a cycle guard. Targets excluded through exclude_files (used by forget) are not followed, so forget gives the same result as a build without those files. Results are cached per parser instance, and exclude_files clears the cache.
  2. parser.py: Python as aliases keep the original name. from m import a as b; b() was qualified as m.py::b. The existing JS original-name key now records Python aliases too, so _resolve_call_target asks for a. The prefixed key starts with _, so star-export discovery already filters it out. This is a general fix, not specific to shims.
  3. parser.py and graph.py: module receivers follow re-exports. pkg.helper() where pkg only re-exports helper now stamps receiver_module with the defining file (via the existing _python_reexport_origin). The bare-endpoint resolver accepts a candidate in the receiver's own file as evidence, because the call-site file does not import the defining file directly. Candidates are still narrowed to top-level names in exactly that file first, so this does not loosen the receiver rule.

There is no schema change and no new post-processing pass: the graph resolver needs no knowledge of shims.

Known limitation

Importers now depend on the real module, so update does not re-parse them when only the shim file changes. That is the same dependency rule as every other import. Re-pointing a shim to a different module needs a build. This is noted in the CHANGELOG entry.

How it was tested

New module tests/test_python_module_alias_shims.py (22 tests). It covers:

  • a simple shim;
  • a shim of a shim;
  • a re-export through a module receiver, with and without a shim;
  • from shim import x and from shim import x as y, plus a plain as alias;
  • TESTED_BY;
  • a missing name that stays bare;
  • four swaps that must not be followed (computed, relative, conditional, outside the repository);
  • a shim cycle;
  • review follow-ups: several swaps (the last one wins), a literal swap overwritten by one that can't be followed, from shim import * direct and recursive, a renamed re-export next to an unrelated function of the same name, and a later unaliased import that rebinds an alias;
  • build/update parity: a post-processing pass run twice is a no-op, and after a new function is added to the real module, incremental_update resolves an untouched caller's shim.new() and matches a fresh full_build.

Ten of the 16 fail on staging without the change. The six that pass there are the guard cases.

uv sync --extra dev
uv run pytest tests/ --tb=short -q
# Run with five modules split out (see note below):
# 4802 passed, 880 skipped, 3 xfailed, 2 xpassed
# Each of those five on its own: watch_refresh_without_index 5, watch_robustness 61,
# postprocessing 20, embedding_refresh_skip_notice 30, incremental 139: all passed
uv run ruff check code_review_graph/
# All checks passed!
uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
# Success: no issues found in 77 source files
bandit -r code_review_graph/ -c pyproject.toml   # via uvx --from 'bandit[toml]'
# No issues identified.

Note on the full run. Locally, uv run pytest tests/ sometimes stops partway with a KeyboardInterrupt raised from unittest/mock.py. A test in one of test_watch_refresh_without_index.py, test_watch_robustness.py, test_postprocessing.py, test_embedding_refresh_skip_notice.py or test_incremental.py lets a mocked KeyboardInterrupt escape. Running those five modules together reproduced it 3 times in 4 on staging without this change, so it predates this PR. Each module passes on its own, and the rest of the suite passes with them excluded.

tests/test_python_star_imports.py: the two read-count tests now expect the helper module to be read twice per process instead of once. One read checks whether it is a shim and one computes its star exports. Both are cached process-wide, so concurrent parsers still read it only twice in total.

Checklist

  • Tests added for new functionality
  • All tests pass: uv run pytest tests/ --tb=short -q
  • Linting passes: uv run ruff check code_review_graph/
  • Type checking passes: uv run mypy code_review_graph/ --ignore-missing-imports --no-strict-optional
  • Lines are at most 100 characters
  • Docs updated where behavior changed (README, docs/, docstrings): CHANGELOG Unreleased (Added and Fixed) and docstrings

🤖 Generated with Claude Code

A compatibility shim that does
`sys.modules[__name__] = importlib.import_module("pkg.real")` hands its
importers `pkg/real.py`, but the parser stopped at the shim: the import
edge named the shim, `shim.func()` and the TESTED_BY edge from a test
calling it stayed bare (`func` is not top-level in the shim), and
`from shim import func` qualified as a node that does not exist.

Python module resolution now maps a shim to its target, through chains
of shims, so the import edge, module-receiver evidence, from-imports and
re-export origins all see the real module. Only the literal,
module-level form is followed; detection is cached process-wide by
mtime and size like the star-export cache.

Two general fixes the shim cases depend on:
- `from m import a as b; b()` resolves to `m.py::a`, not `m.py::b`.
- `pkg.name()` where `pkg` only re-exports `name` stamps the defining
  file as `receiver_module`, and the bare-endpoint resolver accepts the
  receiver's own file as evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 01:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved wildcard imports and incorrect targets from rebinding, renamed re-exports, and multiple module replacements can produce inaccurate graph edges.

4 open issues
What changed in this PR

This PR improves Python dependency tracking by following sys.modules compatibility shims to their installed modules, so imports and calls can resolve to actual definitions.

Changes:

  • Adds bounded, cached shim resolution and preserves original names for Python import aliases.
  • Follows module-receiver re-exports and accepts their defining files during graph resolution.
  • Adds regression coverage and documents the rebuild requirement after retargeting a shim.
File Description
tests/​test_python_star_imports.py Updates read-count expectations for cached shim detection.
tests/​test_python_module_alias_shims.py Adds shim, alias, receiver, guard, and build/update regression tests.
code_review_graph/​parser.py Adds shim detection and updates import and receiver resolution.
code_review_graph/​graph.py Accepts receiver-module evidence when resolving bare endpoints.
CHANGELOG.md Documents resolution improvements and incremental-update limitations.

Review effort: Balanced

Comment thread code_review_graph/parser.py Resolved
Comment thread code_review_graph/parser.py Resolved
Comment thread code_review_graph/parser.py Resolved
Comment thread code_review_graph/parser.py Resolved
- `from shim import *` enumerates the real module: star-import
  expansion now resolves through `_python_import_origin`, which follows
  shims, so direct and recursive wildcard imports reach the definition.
- Shim detection takes the last write to `sys.modules[__name__]`, as
  importers do, and declines when that write is not a literal swap or
  when any write sits inside a block.
- A renamed re-export (`from .impl import charge as pay`) keeps `pay` on
  the re-exporting module instead of sending `pkg.pay()` to an unrelated
  `impl.pay`.
- A later unaliased `from b import bar` drops the original-name entry an
  earlier `from a import foo as bar` left behind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 01:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shim detection can mistake a locally defined callable for importlib’s function and redirect graph edges to an unrelated module.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Reject shadowed import_module callees in module swap detection

code_review_graph/​parser.py:15157

The bare-call branch accepts any function named import_module, even a local function rather than the importlib function. For example, def import_module(name): return sys.modules[__name__] followed by sys.modules[__name__] = import_module("target") leaves the original module installed at runtime. This detector instead follows target when that file exists in the repository, attributing import and call edges to an unrelated module. Verify the callee's import binding and reject shadowed or rebound callees before following a swap; apply the same check to the importlib.import_module form. Add a guard regression for a locally defined wrapper while preserving support for from importlib import import_module.

4 resolved since last review

Review effort: Balanced

…dule

Shim detection accepted any callable spelled `import_module` or
`importlib.import_module`, so a module defining its own
`def import_module(...)`, or rebinding `importlib`, was read as a shim
and its importers redirected to an unrelated module. The callee name
must now be bound only by an import from importlib (`import importlib`,
`import importlib.util`, or `from importlib import import_module`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 23, 2026 01:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

External symlink reads, incorrect shim resolution cases, and Windows-incompatible test paths remain unresolved.

1 open issue
3 previously missed issues

In code that hasn't changed since last review

Medium severity Require an unshadowed sys import before redirecting edges

code_review_graph/​parser.py:15169

The detector validates importlib but never checks what sys refers to. With import fake_sys as sys, where fake_sys.modules is an ordinary dictionary, the assignment does not replace the importing module, yet the detector redirects its import and call edges to the named target. Require an unshadowed sys import before accepting the swap, and add a regression for a shadowed or rebound sys.

Medium severity Resolve shim submodule dependencies to the installed package

code_review_graph/​parser.py:15804

Submodule dependencies still use the shim's original name when the shim installs a package. For example, if shim.py installs pkg, _python_package_dir now finds pkg, but _python_from_import_targets emits shim.child for from shim import child. Resolution then searches for shim/child.py, not the existing pkg/child.py, and leaves that dependency unresolved. Importer queries and incremental dependency tracking therefore miss the child file. Resolve these submodule targets relative to the installed package and add a shim-to-package regression that checks the IMPORTS_FROM edge.

Medium severity Use POSIX paths for graph identity comparisons

tests/​test_python_module_alias_shims.py:102

Graph paths use forward slashes on every platform (normalize_file_path in parser.py:911–925), but str(file_path.resolve()) uses backslashes on Windows. This filter therefore returns no edges, so the new positive regression tests fail even when shim resolution is correct. Use .resolve().as_posix() for graph identities throughout this test module, including _paths() and the expected qualified names in individual tests.

Review effort: Balanced

Comment thread code_review_graph/parser.py Resolved
- Read a module for shim detection only when its resolved path is inside
  the repository, so a symlink to an outside file is never opened.
- Require `sys` to be bound only by `import sys`; `import x as sys` or a
  rebinding makes `sys.modules[__name__] = ...` replace nothing.
- `from shim import child`, where the shim installs a package, depends
  on that package's `child` file: the target is the submodule's file
  when the dotted name would point under the shim, and Python module
  resolution accepts an absolute in-repository path.
- Tests compare graph identities as POSIX paths, so they hold on
  Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacksonFuck

Copy link
Copy Markdown
Contributor Author

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because re-export resolution can bypass repository boundaries and read external files through symlinks, alongside alias-resolution and rebinding concerns.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

tirth8205 added a commit that referenced this pull request Nov 6, 2026
Legend icons were stretched to 104px by the page-wide svg rule, so the legend
filled the screen and a sixth node kind pushed it under the filter panel. This
is older than the train, and the icons are now 16px. Two tests also depended on
their platform: Python 3.10 accepts the deep expression (#1133), and Windows
paths have backslashes (#1108).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tirth8205 tirth8205 closed this pull request by merging all changes into tirth8205:staging in 0160eb0 Nov 6, 2026
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…irth8205#1133)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
…rts (review of tirth8205#1133)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gowrishacv pushed a commit to gowrishacv/code-review-graph that referenced this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants