Skip to content

Post step: defer mirror repacks until 50 small packs, cutting the post step from 14s to 3s on large mirrors - #67

Draft
jahands wants to merge 2 commits into
mainfrom
defer-small-pack-rollup
Draft

jahands wants to merge 2 commits into
mainfrom
defer-small-pack-rollup

Conversation

@jahands

@jahands jahands commented Nov 11, 2026 •

Copy link
Copy Markdown

Problem

After every sync, the post step runs git repack --geometric=2 on the mirror. On a large mirror that repack takes about as long however little the sync fetched, and every job that commits pays it. In a full CI run of a large open-source monorepo (12.6 GiB mirror, ~220 jobs), the post step averaged 14.4s and reached 56s.

Change

Skip the repack until 50 small packs have built up, which is git's gc.autoPackLimit default. The rest of mirror maintenance is unchanged.

Results

Before After
Post step mean, full CI run 14.4s 3.3s
Post step max, full CI run 56s 7s
Jobs that repacked, one day of CI all 37k committing jobs 612
Post-step time, one day of CI ~149 job-hours ~36 job-hours
  • CI runs: before and after are separate runs of the repo's CI on a fork.
  • One-day rows: the repo's CI jobs from 2026-11-10, replayed through a model of the mirror's snapshots. The post-step totals apply the CI runs' per-job means to those jobs.

Trade-off

Once every ~50 snapshots, the jobs that start from that snapshot still pay a repack, as every job does today. That is the 612 jobs above, 18 times in the day.

Tests

  • New: maintenance skips the repack below the limit and repacks at it.
  • Updated mirror-cleanup and mirror-gc-git tests for the limit.

Created with Claude Opus 5.5 in Claude Code.


Watch Proof live
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need. (Staging)

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ade2fa3. Configure here.

Comment thread src/blacksmith-cache.ts Resolved
@blacksmith-staging

blacksmith-staging Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Matches Post step skips mirror repack below 50 small packs

With the rebuilt dist, the post step still skips the repack below 50 small packs (about 3s down to about 0.16s per job under 8-way load) and still rolls up at 50, leaving the same refs and objects as before.

Merge-base

Post step under load

This PR 6217720a

Post step under load

Roll-up at 50 packs

Rolled-up mirrors match

Caution

New issue found while testing

Large sync packs never reach the roll-up limit

With 55 sync packs of 10 MiB, deferral leaves 30 small packs, so every post step skips the repack and the mirror stays at 61 packs. Before, three jobs folded it to 4 packs.

Merge-base

Large packs, before

This PR 6217720a

Large packs, after

Proof on 6217720a (from ade2fa32) · mocked: Sticky-disk Git mirror (Each job gets a local copy of a public repo's bare mirror with synthetic sync pa...) · not run: Blacksmith VM agent (sticky disk gRPC) (Only exists on Blacksmith runners; the post step's commit call fails after maint...) · Did Proof get it right? Yes · No

View full Proof report · React 👎 or comment @codesmith dismiss this proof to dismiss

@jahands jahands changed the title Post step: defer mirror roll-ups until 50 small packs (post step 20s → 3s under load) Post step: defer mirror repacks until 50 small packs, cutting the post step from ~15s to ~3s on large mirrors Nov 11, 2026
@jahands jahands changed the title Post step: defer mirror repacks until 50 small packs, cutting the post step from ~15s to ~3s on large mirrors Post step: defer mirror repacks until 50 small packs, cutting the post step from 14s to 3s on large mirrors Nov 11, 2026
@jahands
jahands marked this pull request as draft November 11, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant