Distribution gate: install the built wheel and sdist, then run them by tirth8205 · Pull Request #1012 · tirth8205/code-review-graph

tirth8205 · 2026-09-16T21:40:42Z

Why

CI installs this project with pip install -e .. That reads every file
straight out of the checkout and never runs the build backend, so nothing in
the suite can catch a file that fails to reach the wheel.

That is exactly how #909 happened: the shipped skills were missing from the
wheel while tests/test_qoder_bundled_skills.py stayed green, because
install_qoder_skills falls back to Path(__file__).parent.parent / "skills"
and under an editable install that fallback resolves to the checkout's own
skills/ directory. The test could not tell a working wheel from a broken one.

What this adds

tests/test_packaging.py, marked packaging and skipped by default.

Builds both artefacts with python -m build into a temp directory.
Installs the wheel into a fresh virtual environment and drives the
installed program from a working directory unrelated to this checkout:
--version, build over a small real repository assembled from this
project's own source files, status --json, detect-changes --base HEAD~1,
visualize (the only path that reads the vendored D3 asset), and the MCP
server over stdio spoken to with the real mcp client
(initialize, tools/list, prompts/list, and a tools/call that reads
the packaged LLM reference document).
Does all of the above again for the sdist, since building from source is
a different path.
Asserts the wheel's contents explicitly. The list of required data files is
derived by parsing every packaged module and statically evaluating its
importlib.resources.files("code_review_graph") and Path(__file__).parent
expressions, rather than by trusting the pyproject include list. It resolves
code_review_graph/assets/d3.v7.min.js,
code_review_graph/docs/LLM-OPTIMIZED-REFERENCE.md,
code_review_graph/eval/configs/*.yaml and the
code_review_graph/_bundled_skills resource, and requires the bundled skill
set to equal the checkout's skills/ tree exactly.
Checks the metadata a user sees: version agrees across __init__,
pyproject.toml, the wheel METADATA and both filenames; the declared
Python floor matches pyproject, parses with ast(feature_version=...), and
is installed onto a real interpreter of that version where every packaged
module is imported; both console scripts exist, import and are callable;
and the core path runs with no optional-extra distribution installed.
Asserts the wheel ships no tests, dot-directories, agent config files,
databases or scratch trees, and that the sdist's only dot-path is the
.gitignore hatchling cannot suppress.

Every smoke test first asserts the installed code_review_graph resolves
inside the environment's site-packages and not under REPO_ROOT, so a
pass can never come from the source tree leaking in.

Canary

test_discovery_finds_the_known_data_surface guards the derivation the
inventory check depends on. If the static evaluator stops finding path
expressions, the inventory test would assert nothing and pass; the canary fails
instead, naming what it did and did not find.

Proof of teeth

Each class of check was broken and the failure recorded, then restored.

Removed [tool.hatch.build.targets.wheel.force-include] from pyproject,
reproducing watch ignore snapshot, partial-overlap root guard, and Qoder skills missing from wheel installs #909:

AssertionError: the wheel has nothing under 'code_review_graph/_bundled_skills',
but packaged code resolves that directory as a package resource
AssertionError: bundled skills in the wheel do not match the checkout's skills/ tree.
AssertionError: None  (install_qoder_skills returned None from the wheel install)

Force-included a tests/test_leak.py and a .beads/beads.db into the wheel,
and bumped __version__ to 2.4.0:

AssertionError: the wheel ships files that must never reach a user:
['.beads/beads.db', 'tests/test_leak.py']
AssertionError: pyproject version 2.3.8 != __init__ 2.4.0

Moved code_review_graph/assets/d3.v7.min.js out of the package:

AssertionError: the vendored D3 asset did not reach the generated page,
so the wheel either omits code_review_graph/assets/d3.v7.min.js or ships it corrupt

Numbers

wheel code_review_graph-2.3.8-py3-none-any.whl: 658,602 bytes, 92 files
sdist code_review_graph-2.3.8.tar.gz: ~670,000 bytes, 107 files
Nothing shipped in the wheel that should not be. Its file list is exactly the
git-tracked contents of code_review_graph/ plus the seven bundled
SKILL.md files and dist-info.
Gate runtime: about 75 seconds with a warm pip cache.

Running it

uv run --python 3.13 python -m pytest tests/test_packaging.py -m packaging

Skipped by a hook in tests/conftest.py unless the marker expression names it
or CRG_RUN_PACKAGING_TESTS=1 is set, so pytest tests/ is unaffected.

Gates

uv run --python 3.13 python -m pytest tests/ -q — 3383 passed, 26 skipped, 2 xpassed
uv run ruff check code_review_graph/ — clean (and on the new test files)
uv run --with mypy mypy code_review_graph/ --ignore-missing-imports --no-strict-optional — clean
pytest tests/test_packaging.py -m packaging — 16 passed

Findings, not fixed here

Reported separately; no production code is changed by this PR.

🤖 Generated with Claude Code

CI installs with `pip install -e .`, which reads every file straight out of the checkout and never exercises the build backend. Nothing in the suite could catch a file that fails to reach the wheel; that is how the shipped skills went missing from the wheel in #909 while `test_qoder_bundled_skills` stayed green, because its editable-install fallback resolved to the checkout's own `skills/` directory. tests/test_packaging.py builds a wheel and an sdist with `python -m build`, installs each into its own throwaway virtual environment, and drives the installed program from a working directory unrelated to this checkout: `--version`, `build` over a small real repository assembled from this project's own sources, `status --json`, `detect-changes`, `visualize` (the only reader of the vendored D3 asset), and the MCP server over stdio spoken to with the real `mcp` client. Each smoke test first asserts the installed package resolves inside the environment's site-packages and not under the repository root, so a pass can never come from the source tree leaking in. The inventory of data files the wheel must carry is derived from the code rather than from the pyproject include list: every packaged module is parsed and its `importlib.resources.files("code_review_graph")` and `Path(__file__).parent` expressions are statically evaluated. `test_discovery_finds_the_known_data_surface` is the canary for that derivation, so a broken evaluator fails loudly instead of leaving the inventory check asserting nothing. Also checked: the wheel carries no tests, dot-directories, agent config files or databases; the sdist's only dot-path is the `.gitignore` hatchling cannot suppress; version agrees across `__init__`, pyproject and both artefacts; the declared Python floor both parses and installs on a real interpreter of that version; entry points import and are callable; and the core path runs with no optional-extra distribution present. The marker is registered in pyproject and skipped by default by a hook in tests/conftest.py. Run it with: uv run --python 3.13 python -m pytest tests/test_packaging.py -m packaging Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-16T21:41:43Z

code-review-graph review

Overall risk: 0.40 (MEDIUM) — 40 changed function(s)/class(es), 0 affected flow(s), 15 test gap(s)

Risk-scored changes

Risk	Level	Symbol	Location	Tested
0.40	medium	tests/test_packaging.py::_static_str	tests/test_packaging.py:255	no
0.40	medium	tests/test_packaging.py::_static_path	tests/test_packaging.py:263	no
0.35	low	tests/test_packaging.py::_bin_dir	tests/test_packaging.py:127	no
0.35	low	tests/test_packaging.py::InstalledEnv	tests/test_packaging.py:142	no
0.35	low	tests/test_packaging.py::Artefacts	tests/test_packaging.py:182	no
0.35	low	tests/test_packaging.py::_module_string_constants	tests/test_packaging.py:239	no
0.35	low	tests/test_packaging.py::_json_tail	tests/test_packaging.py:950	yes
0.30	low	tests/conftest.py::pytest_collection_modifyitems	tests/conftest.py:17	no
0.30	low	tests/test_packaging.py::_clean_env	tests/test_packaging.py:85	yes
0.30	low	tests/test_packaging.py::wheel_names	tests/test_packaging.py:203	no

Test gaps

tests/conftest.py::pytest_collection_modifyitems (tests/conftest.py:17)
tests/test_packaging.py::_bin_dir (tests/test_packaging.py:127)
tests/test_packaging.py::InstalledEnv (tests/test_packaging.py:142)
tests/test_packaging.py::_install_into_fresh_venv (tests/test_packaging.py:152)
tests/test_packaging.py::builder_python (tests/test_packaging.py:172)
...and 10 more without direct tests

Token savings: this graph-backed report used ~10,892 fewer tokens (~65%) than reading every changed file in full (estimated, chars/4 approximation).

Powered by code-review-graph — local-first analysis; no code leaves the CI runner.

The four release checks built as PRs tirth8205#1012, tirth8205#1013, tirth8205#1014 and tirth8205#1016 each run on demand and gate nothing. Alongside them, the end-to-end MCP client tests, the browser tests and the coverage matrix already run on every pull request but are not required checks, so nothing they say has ever stopped a promotion. This wires all seven into one workflow that runs when code lands on `testing`, which is the branch a release is cut from. Not on pull requests. Every job here takes minutes and a contributor must not wait for them; ci.yml is untouched. What blocks and what only reports is decided per check and justified in CONTRIBUTING.md. Blocking: upgrade-path (a database a user already has is not ours to lose), packaging (a broken wheel reaches everyone at once), determinism (no network, so a failure is a real difference), the coverage matrix, and the end-to-end tests on Linux and macOS. Reporting: the pinned real-repository corpus (it clones eight repositories that belong to other people, and a failed clone must not stop a release), the browser tests, and the Windows end-to-end leg. Reporting is not ignoring. scripts/promotion_gate.py wraps every check and fails it when the check did not actually run: its test module is not in the checkout, the selection collected almost nothing, or the tests were skipped by an opt-in hook instead of executed. Three of the four new checks are opt-in through a pytest_collection_modifyitems hook in tests/conftest.py and those hooks arrive on four branches that overwrite one another on merge, so a bare `pytest -m corpus` step exiting 0 on twenty skipped tests is a real prospect, not a hypothetical. The result is posted once to the job summary and once to the open issue labelled promotion-gate, naming every check and, for a failure, the assertion that moved. The gate opens no pull request and merges nothing. tests/test_promotion_gate.py asserts the policy, drives each canary to failure with a real junit report, and checks the workflow against the registry so a check cannot be added to one and forgotten in the other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This was referenced Sep 16, 2026

ci: one promotion gate on testing, with a blocking policy that is tested #1020

Merged

Integrate the release gates: 11 checks covering upgrade, packaging, determinism, the MCP surface, the CLI, exports, concurrency, the Action and the corpus #1024

Merged

tirth8205 closed this pull request by merging all changes into staging in 14fd8a4 Sep 16, 2026

tirth8205 deleted the test/packaging branch September 18, 2026 12:59

Navigation Menu

Repository navigation

Distribution gate: install the built wheel and sdist, then run them - #1012

Distribution gate: install the built wheel and sdist, then run them#1012
1 commit merged into
stagingfrom
test/packaging

tirth8205 commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tirth8205 commented Sep 16, 2026

Why

What this adds

Canary

Proof of teeth

Numbers

Running it

Gates

Findings, not fixed here

Uh oh!

github-actions Bot commented Sep 16, 2026

code-review-graph review

Risk-scored changes

Test gaps

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant