Skip to content

Benchmark examples: replace uv pip in the Docker wrappers and matrix helpers with uv run - #502

Merged
tae898 merged 2 commits into
mainfrom
fix-393-uv-run
Oct 24, 2026
Merged

tae898 merged 2 commits into
mainfrom
fix-393-uv-run

Conversation

@tae898

@tae898 tae898 commented Oct 24, 2026

Copy link
Copy Markdown

Closes #393.

Follow-up to #381 and PR #392. The benchmark examples no longer use uv pip or a hand-made venv.

  • Examples 07 to 12, Docker wrappers: python -m venv + activate + uv pip install is replaced by one uv run --no-project --python <interpreter> --with <package> ... python bench_exec.py <script> <args>. Every package and every pin is unchanged (ladybug==0.19.0 stays). The ArcadeDB wheel is a --with of the same call. The unused python3-venv apt package is dropped from the Debian-based branches (the postgres and pgvector containers), and the uv bootstrap there now runs with python3 because there is no venv python any more.
  • New bindings/python/examples/bench_exec.py: prints one [bench-env] python=... line and one [bench-env] name==version line per installed package (importlib.metadata, the pip freeze equivalent), then os.execvs python -u <script>. The measured process is the interpreter that printed the versions, in the same environment, so the log of every run records what it ran with.
  • Milvus runner image (11 and 12): uv pip install --system is replaced by a build-time uv run --with numpy --with pymilvus --with psutil that fills UV_CACHE_DIR=/opt/uv-cache, and the runs use uv run --offline against that cache, so a run cannot resolve other versions than the image holds (as before, the versions are frozen when the image is built). The image tag is bumped to py312-v2 so an existing v1 image is not reused.
  • examples/scripts/_matrix_helpers.sh: the local wheel is no longer installed into the host environment or the user site. matrix_prepare_local_arcadedb_wheel sets MATRIX_PYTHON=(uv run --no-project --python <python3> --with <wheel> python3), the six run_*_matrix.sh scripts start the launcher with it, and the helper logs the launcher's python, arcadedb-embedded and jpype1 versions. uv run --with layers over the active virtual environment (checked), so the launcher still sees the dependencies of that environment, such as lxml.
  • benchmarks/experiments/CAMPAIGN.md: the uv pip install --python .venv <wheel> jpype1==1.7.1 note now says uv add jpype1==1.7.1 at the repo root (not run here; it records the pin in pyproject.toml and uv.lock). Docs: the six docs/examples/07 to 12 pages and ci-setup.md.

Same versions before and after, in python:3.12-slim with lxml duckdb arcadedb-embedded==26.10.2 (both commands built from the real uv_bootstrap_commands and uv_run_command): the old flow (python -m venv, uv pip install, python bench_exec.py) and the new flow print identical [bench-env] lines, which is the same CPython 3.12.15, lxml 6.1.3, duckdb 2.0.0, arcadedb-embedded 26.10.2, jpype1 1.7.1 (and the same packaging and pip), and the probe script ran in both. Also run: the Debian bookworm-slim + apt python3 path (the postgres and pgvector pattern, no python3-venv), where uv run --python python3 --with lxml --with psycopg[binary] printed python 3.11.2, lxml 6.1.3, psycopg 3.3.6; and the Milvus pattern (cache filled by one uv run as root, then uv run --offline as uid 4242 from that cache) printed numpy 2.5.4, pymilvus 3.0.2, psutil 7.2.2. All containers were pinned to cores 0 to 3 with a 2 GB limit.

No speed claim: this changes how the environment is built, not what the measured process executes, and no timed run was made. The interpreter start of the measured process is now uv run then execv instead of the venv python, outside the script's own timers. The issue asked for a short tools/ab.py --noise round per changed example if a measured phase includes interpreter start; that is left to the benchmark queue (none of the examples' timers include it), so I did not run it.

Checks run: pre-commit (black, isort, shfmt and the rest) on every changed file, again after the formatter's own edits; bash -n on _matrix_helpers.sh and the six run_*_matrix.sh; shellcheck (same 9 findings with and without the change, none new); py_compile of 07 to 12; pytest on the new test_bench_exec.py (the environment print, the exec into the script, and a guard against uv pip and venvs in the six wrappers) and the existing test_example11_degree_matching.py (5 passed, run against the published 26.10.2 wheel). Not run: the full wrappers against a real dataset, the Milvus runner image build itself, and _matrix_helpers.sh against a freshly built wheel (needs the Linux wheel build); CI covers the examples.

Remaining uv pip outside this scope, not touched: e2e-python/README.md (upstream's suite), docs/superpowers/plans/2026-07-08-bolt-ci-nightly-matrix-4891.md and docs/superpowers/specs/2026-07-08-bolt-ci-nightly-matrix-4891-design.md (historical plans). The ci-setup.md sentence that says there is no uv pip in the workflows is a statement, not a command.

🤖 Generated with Claude Code

tae898 and others added 2 commits October 24, 2026 20:25
…helpers with uv run

The Docker wrappers of examples 07 to 12 run the example with uv run --no-project --with (no venv, no uv pip), through bench_exec.py, which prints the interpreter and every installed package version before it becomes python -u <script>. The Milvus runner image fills its uv cache at build time and runs offline. _matrix_helpers.sh no longer installs the wheel into the host environment; the matrix launcher runs under uv run --with the wheel.

Closes #393

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Bandit flags the test's own subprocess call; the argv is fixed
(sys.executable plus a repo script), no shell, no untrusted input.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 2c53a45 into main Oct 24, 2026
43 of 45 checks passed
@tae898
tae898 deleted the fix-393-uv-run branch October 24, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark examples: replace uv pip in the Docker wrappers of examples 07-12 and _matrix_helpers.sh

1 participant