Repository navigation
v1.91.283.0 fix(benchmark): reconcile the collection, comparison, budget and trend phases with the vet-the-number section (GSTA-128) - #3520
Merged
Conversation
…d phases with the vet-the-number section The perf-collection judge (run 40558235014, both attempts, six samples) named the same contradictions: Phase 3 offered an optional 3-run median while the mandatory vet section demands 5 alternating runs per side and a stored baseline cannot be interleaved; the budget example read '4/6 passing' against a rule that counts WARNING as below budget; the trend table showed Grade and Bundle columns no stored file defined; Phase 9 named two files with no contents. Phase 3 now runs 5 per page with median and range and names --quick as the one-run ballpark; the baseline schema carries runs and range_ms; Phase 5 says how the seven questions apply against a stored baseline (compare the delta with both spreads, name the limiter from Phase 6, inconclusive when either side is one run); the grade example counts 3 PASS, 1 WARNING, 2 FAIL; Phase 8 reads the benchmark and baseline JSON files by timestamp; Phase 9 defines the JSON as the Phase 4 object plus grade, verdict and per-metric status. Nearby prose tightened so the skeleton stays under its 31,300-byte cap (31,290). Judge after the repair: clarity, completeness and actionability 4/4/4 on all three samples, twice. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…on (GSTA-128) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
Step 14.5 of /ship on #3520: the /benchmark section said 'Multiple runs averaged' and '3 runs each'; it now matches Phase 3 (five runs per page, median with range, --quick a single-run ballpark). GSTA-128. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… base=origin/main@9ccb523 policy=base origin/main@9ccb5233f309:.gstack/ship-policy.json) ship, next in queue at 1.91.283.0 STAMP_VERSION: 1.91.283.0 claims: ignored (stamp-at-merge) stage version: done (wrote 1.91.283.0) stage release_tool: done (bun scripts/gen-agents-digest.ts) stage changelog: done (Unreleased -> 1.91.283.0) stage stamp_paths: skipped (no tentative mention newer than the fork version 1.91.281.0 and none of base 1.91.282.0 on a line this branch added) stage mirror: skipped (no mirror in policy) stage apply: done (4 file(s)) diff --git a/CHANGELOG.md b/CHANGELOG.md index 695e20b..15162a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [1.91.283.0] - 2026-10-27 **/benchmark's collection, comparison, budget and trend phases now agree with its vet-the-number section.** The 2026-10-27 flaky pass (GSTA-128) read the `benchmark/SKILL.md perf collection` judge's six samples from run 40558235014: every sample named the same contradictions, so the prose was repaired and no threshold moved. diff --git a/VERSION b/VERSION index 269b6e8..0b68a05 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.282.0 +1.91.283.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 583844b..894898d 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.282.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.283.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/node_modules b/node_modules new file mode 120000 index 000000000..858ae29 --- /dev/null +++ b/node_modules @@ -0,0 +1 @@ +/workspace/gstack/node_modules \ No newline at end of file diff --git a/package.json b/package.json index 66ec462..da0b872 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.282", + "version": "1.91.283", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", FILES: CHANGELOG.md VERSION agents-digest/gstack-AGENTS.md package.json MERGE_CONDITION: sha=aa8c07b76935d91ec548fb5a32fd429392f88460 next: NEW_VERSION=1.91.283.0; commit and push only the stamp diff above (FILES), with no edit of your own: the train's verify accepts the CHANGELOG only as the base's plus this PR's entry re-headed; merge with the expected head aa8c07b and recompute on a stale queue. GSTACK_RESULT: skill=restamp status=complete run=/workspace/gstack/.gstack/tmp/restamp-journal.json Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Phase 5 before (main) and after (this branch): the comparison now opens with a verdict, names the run count, and says what a stored-baseline comparison can and cannot claim.
Before/after as text: Phase 5 now says how the seven vet-the-number questions apply against a stored baseline: compare each delta with both sides' spread, name the limiter, and call one-run comparisons inconclusive
Daily flaky pass (GSTA-128) follow-up for the
benchmark/SKILL.md perf collectionjudge case, which failed twice on the #3518 screen (run 40558235014, clarity 3 < 4, completeness 3 < 4, actionability 3 < 4, 3 samples each) on bytes train 25's gate had judged 24/24 two hours earlier.Diagnosis. All six judge samples name the same contradictions in Phases 3–9, so the case sits at its threshold on real prose conflicts, not judge noise:
Repair (
benchmark/SKILL.md.tmpl, regeneratedSKILL.md; the vet-the-number section is untouched):--quickis named as the one-run ballpark with no comparison verdict.runsandrange_ms.runscounts as one). A delta smaller than either spread is OK whatever threshold it crosses. The report leads with the verdict and names the run count.timestampand says where Bundle and Grade come from; Phase 9 defines the JSON (Phase 4 object plus per-pagegrade,verdictand metricstatus) and uses the same UTC-timestamp filename as Phase 4.docs/skills.md: the /benchmark section said "Multiple runs averaged" and "3 runs each"; it now says five runs per page, median with range,--quicka single-run ballpark.Why
The vet-the-number section (absorbed from pstack, #3487) made the skill contradict itself on run counts and on what a stored-baseline comparison can claim; a judge reading the whole bundle has to pick a side. This resolves the demonstrated ambiguities together in the skill's own phases instead of adding clauses.
Verification
EVALS=1 EVALS_TIER=gate … bun test test/skill-llm-eval.test.ts -t 'benchmark/SKILL.md perf collection'): clarity, completeness, actionability 4/4/4 on all three samples, four runs ($0.06 each): three on the repaired tree before the commit, one by /ship on the committedbenchmark/SKILL.mdbytes (sha256 15ac49f4, unchanged by 7bcd06e).Approvals
Standing: CREW.md rule 16 (daily flaky pass, Garry 2026-10-19); GStack Minder's go for this follow-up (2026-10-27). Lane paid spend $0.24 of the $10 cap (four judge runs); Ubicloud $0.14.
Blast radius
Two prose files and one doc:
benchmark/SKILL.md.tmpl, its generatedSKILL.md(bun run gen:skill-docsran at both heads, tree clean, 31,290 bytes), and the /benchmark paragraph ofdocs/skills.md. The vet-the-number section, the carve guard entry, every test and every other skill are unchanged (gate scan: 0 eligible hunks). Behavior change for users of /benchmark: five runs per page by default instead of one (the section already required it before any comparison was reported), and--quickkeeps the single run. Rung: script ran (regen, guards, judge, remote free gate).Summary
The
benchmark/SKILL.md perf collectionjudge read Phases 3–9 and found the skill arguing with its own vet-the-number section about run counts and what a stored-baseline comparison can claim. The phases now say five runs, medians with ranges, and a verdict that calls one-run comparisons inconclusive; the judge rates the bundle 4/4/4 four times running. Commits: the template repair, the CHANGELOG entry, and the docs/skills.md correction from /ship's documentation audit.Test Coverage
Skipped audit: prose-only diff (benchmark/SKILL.md.tmpl, generated SKILL.md, CHANGELOG, docs/skills.md). Phases 3–9 are [→EVAL] paths covered by the judge case 'benchmark/SKILL.md perf collection' (test/skill-llm-eval.test.ts:630, spans Phase 3 to Important Rules). No new application code paths to audit.
Tests: 1501 → 1501 (+0 new)
Coverage gate: could not determine percentage — skipping.
Tests ran: remote-free-gate 18028 · free-tests.yml macos-named-regressions (CI run 40575772087) · windows-free-tests.yml windows-free-shard (CI run 40575772146) · llm-judge perf collection 1
Pre-Landing Review
No issues found. Core checklist clean; specialists testing, maintainability, security, performance, simplification: NO FINDINGS; state-machines skipped (scope); Red Team not activated (144 lines, no CRITICAL).
REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen
Gate edits: none detected (0 listed, 0 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)
Outside review: Codex CLI not installed; the outside voice was a Capy subagent on openai/gpt-6.1-sol (adversarial pass, NO FINDINGS, "Recommendation: proceed"). Native adversarial pass (anthropic/claude-fable-5-1): nine INFORMATIONAL findings, all below the rule-23 floor, noted below and fixed in no pass.
Review notes
Native adversarial findings on
benchmark/SKILL.md.tmpl, kept as notes (none would fail a test, break production or lose data; six ask for design beyond the judged contradictions, two concern example content the diff only tightened; the judge passed these bytes 4/4/4 four times). A follow-up prose pass is the owner's call; any such edit must re-tighten under the 31,300-byte cap (10 bytes of headroom).OK (within noise)status.setTimeout(…, 3000); a null LCP stays N/A in Phase 7 (pre-existing handling), and the median rule does not say how a null among five runs is reported.hero-image.webp, the resource the VERDICT example names as the LCP limiter, and names two remedies against the new one-remedy rule.--quickfiles and baseline files with five-run benchmarks and shows neither runs nor branch.--quick"no comparison verdict" (line 109) does not say to skip Phase 5, so a one-run delta table could still print.runsandrange_msand no failure count (line 111).--baseline" (line 150) does not say on which branch.Exploratory QA
Functional smoke, 8 captures in 55 s of a 300 s budget, all pass: skeleton size cap (31,290 ≤ 31,300); gen:skill-docs leaves the tree clean; skill-md-size-guard 2/0; parity-suite 2/0; pstack-sections 8/0; prose lint ok on the template and the CHANGELOG entry; context-budget-ratchet 7/0; the Phase 7 grade example (3 PASS, 1 WARNING, 2 FAIL = 4/6 below budget) matches its table. Report with linked checkpoints 002–008:
.gstack/qa-reports/ship3520-20261027T195340Z/functional/report.md(lane machine). Limits: no live /benchmark run against a page (browser-bound surface outside the smoke budget; Phases 3–9 are judged by the paid eval); test/carve-guards.test.ts does not exist on this tree, so the carve guards were exercised through the size-guard and parity captures.Design Review
No frontend files changed — design review skipped.
Eval Results
benchmark/SKILL.md perf collection (llm-judge, EVALS_TIER=gate): PASS c:4 c:4 a:4, $0.06, 26 s, run id ship3520-judge on the committed bytes (judge log:
~/.capy/work/flaky/ship3520/judge/out.logon the lane machine). Three earlier runs on the same bytes before the commit: PASS 4/4/4 each, $0.06 each (~/.capy/work/flaky/bench/judge-run{,2,3}/). Lane paid total $0.24.Greptile Review
No Greptile comments.
Scope Drift
Scope Check: CLEAN
Intent: reconcile /benchmark Phases 3–9 with the vet-the-number section so the perf-collection judge stops failing on real contradictions
Delivered: benchmark/SKILL.md.tmpl + regenerated SKILL.md, CHANGELOG [Unreleased] entry, docs/skills.md run-count correction
Plan Completion
Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches). Fix: add "Plan: " to the PR body, or run /autoplan.
Issues
Refs #3520
Verification Results
No plan-specific verification items (no plan file); automatic diff-scoped QA above: 8 PASS, 0 FAIL, 0 BLOCKED, 0 NOT RUN.
TODOS
No TODO items completed in this PR.
Documentation
Status: updated. One factual correction in
docs/skills.md, no blockers.Scope: base 392f50d, branch flaky-2026-10-27-bench, mode edit. The committed diff touches
benchmark/SKILL.md.tmpl, its generatedbenchmark/SKILL.mdand the CHANGELOG [Unreleased] entry. There are no staged, unstaged or untracked changes. The audit covered the candidate's 128 inventoried docs. The banned-terms check returned none.Documentation health:
docs/skills.md: updated. The/benchmarksection said "Multiple runs averaged" and showed "3 runs each"; it now says five runs per page, median with range, and--quickas a single-run ballpark.README.md: current. The/benchmarkrow names no run counts.CHANGELOG.md: current. The [Unreleased] entry matches the template change. It was not edited.benchmark/SKILL.md.tmplandbenchmark/SKILL.md: reviewed, not edited (the generated file is already in the diff).Coverage debt:
/benchmarkhas reference coverage inREADME.mdanddocs/skills.md, with no how-to, tutorial or explanation page. That is existing debt, not caused by this change./document-generatecould fill it.Diagram drift: none. The diff touches no ASCII or Mermaid diagram entities.
Test plan
gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commandsat tree 247d5592: 18,028 tests, 0 failed ($0.08); at tree 51899fae: 17,995 / 0 ($0.06)EVALS=1 EVALS_TIER=gate bun test test/skill-llm-eval.test.ts -t 'benchmark/SKILL.md perf collection': PASS c:4 c:4 a:4 ($0.06)bun test test/skill-md-size-guard.test.ts test/parity-suite.test.ts test/pstack-sections.test.ts test/context-budget-ratchet.test.ts: 19 pass / 0 failbun run gen:skill-docs: tree clean;wc -c benchmark/SKILL.md= 31290🤖 Generated with Claude Code