Skip to content

gstack-artifact validate now refuses a run.json that gstack-autoplan never wrote (GST-3090) - #4494

Merged
capy-ai[bot] merged 3 commits into
mainfrom
artifact-validate-refuses-hand-built-runs
Nov 18, 2026
Merged

capy-ai[bot] merged 3 commits into
mainfrom
artifact-validate-refuses-hand-built-runs

Conversation

@garrytan

@garrytan garrytan commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

gstack-artifact validate and ack now refuse a run.json that gstack-autoplan never wrote, so a hand-built run can no longer pass as a finished review.
Why: a session skipped gstack-autoplan next, built run.json with a script copied from the reference fixture, and got ARTIFACT_VALID.
Nothing to do unless you validate old or hand-made run directories: rerun those through gstack-autoplan.

What changed

Before, a hand-built run.json from the reference fixture validated. After, the same file is refused (error lines shortened).

A session that skipped the review built its own run.json from the reference fixture, and the completion check said it was fine. Once run.json parses against its schema, validate and ack now also check what gstack-autoplan writes on every run:

  • ARTIFACT_NOT_AUTOPLAN_RUN: host is not gstack-autoplan, one of the fields gstack-autoplan always writes is missing (plan_input, plan_source, active_plan, restore, out, created_at, revision, attempt_counter, outside_enabled, phase_order, history, scope), or there is no attempts.jsonl journal.
  • ARTIFACT_JOURNAL_MISMATCH: the journal has no init row for the run, a reviewer attempt was never journaled as dispatched, a completed review's output or hash differs from its submitted row, a bound review output changed after submit, or the run's checkpoint is ahead of run.json, disagrees on status or gate revision at the same revision, or holds a journal the live one no longer begins with.

Both codes have troubleshooting entries (docs/troubleshooting.md#artifact-not-autoplan-run, #artifact-journal-mismatch) and are described in docs/unattended.md. Public copies (validate --public) are unchanged, and the library function validateRun stays the structural check; the new check is validateAutoplanRun in lib/autoplan-writer-check.ts.

The reference fixture (test/fixtures/multi-agent-wave/reference-run) was hand-written before gstack-autoplan existed. It stays the structural fixture for validateRun, the CLI now refuses it, and its generator line, docs/unattended.md and INCIDENTS.md say so.

Why

The validator is the completion check every unattended consumer trusts. If a script can produce a file that passes it, a skipped review looks like a finished one.

Blast radius

Only gstack-artifact validate <run.json> and gstack-artifact ack call the new check; gstack-autoplan is the only writer of run.json (lib/autoplan-run.ts:132 sets host: 'gstack-autoplan'), so real runs pass, which the CLI-written control test and the full free test suite in CI show. A run directory written by a gstack older than these fields is refused and has to be rerun; there is no migration for it.

Test Coverage

8 of the 14 paths in lib/autoplan-writer-check.ts have tests that would catch a regression. Untested branches: a wrong host on an otherwise real run, a field of the wrong type, a completed review missing its submit records, a status or gate-revision change at the same checkpoint revision, choosing the checkpoint after resume --from, and the guard that ignores outputs outside the run directory. None is a regression; they are proposed as follow-up tests.

Tests ran: targeted 32
Tests ran: full 1699 files (CI)

Pre-Landing Review

No blocking issues. The specialist review found nothing. The adversarial review raised four notes below the blocking floor, listed under Review notes.

Review notes

Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.

  • INFORMATIONAL lib/result-codes-autoplan.ts:17 check proves self-consistency, not authorship; docs/changelog wording promises provenance
  • INFORMATIONAL lib/autoplan-writer-check.ts:85 a cp -r copy of a run can be falsely refused as checkpoint-ahead when a checkpoint dir is set
  • INFORMATIONAL lib/autoplan-writer-check.ts:79 checkpoint comparison skips silently when no reference generation is found
  • INFORMATIONAL lib/autoplan-writer-check.ts:20 runs begun on an older gstack lacking WRITER_FIELDS fail validate/ack after upgrade; migrate does not backfill
Documentation

Status: updated. One factual correction was made, and there are no blockers.

Scope: 12 committed files from base 782f459 to head 49d5178 on artifact-validate-refuses-hand-built-runs, with no staged, unstaged or new files. These docs were audited against lib/autoplan-writer-check.ts, lib/result-codes-autoplan.ts and bin/gstack-artifact: docs/unattended.md, docs/troubleshooting.md, the changelog.d fragment, test/fixtures/multi-agent-wave/INCIDENTS.md, autoplan/sections/unattended.md.tmpl and autoplan/DRIVER.md.tmpl.

Documentation health

  • docs/troubleshooting.md: updated. The ARTIFACT_JOURNAL_MISMATCH entry now says the checkpoint must also agree on gate revision.
  • docs/unattended.md: current. The validate contract, the ack behavior, the public-copy exemption and the reference-run description match the code.
  • changelog.d/artifact-validate-refuses-hand-built-runs.md: current, not edited.
  • test/fixtures/multi-agent-wave/INCIDENTS.md: current.
  • autoplan/sections/unattended.md.tmpl and autoplan/DRIVER.md.tmpl: current.

Coverage debt: none found. Both new result codes have troubleshooting anchors that match their result-code table entries. Diagram drift: none, because no diagrams are affected.

Tests
  • bun test --timeout=30000 test/artifact-writer-check.test.ts test/headless-artifacts.test.ts: 32 pass, 0 fail. The new file failed first on main 782f459 (7 of 8 failed; the real-run control passed).
  • Full free test suite in CI: Free Tests run 44347505431 at d9b18c1, green.
  • macOS and Windows jobs at d9b18c1: cso-macos-launcher and macos-named-regressions (run 44347505431), windows-smoke (run 44345463484), green.
Agent notes

Bug fix for GST-3090, reported by GEVA-549. No paid runs; the eval selector picked behavioral probes through the bin dependency graph and none ran (paid budget $0).

Refs GST-3090

Approvals

No approvals recorded for branch artifact-validate-refuses-hand-built-runs.
Plan: exempt: bug fix
Plan check: exempt bug fix

REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=4 fixed=0 scope=frozen ledgers_read=2 dispatches=1
Gate edits: none detected (8 listed, 4 read; patterns: RH-1,RH-3,RH-4,RH-12,RH-13,RH-14,RH-15,RH-16; unlisted idioms: none)

Open in Capy

{
  "schema_version": 1,
  "pr": 4494,
  "head": "d9b18c11134f8e6e40acc75488e99f892bb604a9",
  "version": "1.100.0.0",
  "base": "main",
  "gate": {
    "targeted": {
      "pass": 32,
      "fail": 0,
      "skip": 0
    },
    "full": {
      "pass": 1699,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "full",
      "run": 44347505431,
      "attempt": 1,
      "tree": "ce4d3d3198b0c5804074b02090af608f7f4f66cb"
    }
  ],
  "calls": 23,
  "session_kind": "unattended",
  "run": "/home/user/.capy/work/ship-4494",
  "head_tree": "ce4d3d3198b0c5804074b02090af608f7f4f66cb",
  "product_patch_id": "9b74756b67ef239a7e107efd41502cd9b6d64c29",
  "gstack_version": "1.100.0.0",
  "review_depth": {
    "depth": "light",
    "class": "code",
    "runtime_lines": 137,
    "rules": "1afa38a4be3b5eef",
    "base": "fda9dea79f131d7731259b9275fc82c04387ca37",
    "registry": "fda9dea79f131d7731259b9275fc82c04387ca37",
    "gate_scan": {
      "tagged": 0,
      "head": "d9b18c11134f8e6e40acc75488e99f892bb604a9"
    },
    "reasons": [
      "runtime=137"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @ce4d3d3198b0 (8 checks, current for d9b18c11134f)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/44347505431/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/44347505431/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/44345463484/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=ce4d3d3198b0 tests=37844 skipped=105 runs=44347505431/1 profiles=ci-reuse/free-tests heads=d9b18c11134f verified=2026-11-18T16:36:20.222Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "docs/ and README.md audited against the 12 changed files (docs/unattended.md, docs/troubleshooting.md, changelog.d fragment, INCIDENTS.md, autoplan unattended templates)"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 4,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "ledgers_read": 2,
    "dispatches": {
      "specialist": 1
    }
  },
  "gate_dispositions": []
}

…never wrote (GST-3090)

A hand-built run.json with no plan_input and no journal passed validate. validate and ack now also check the fields gstack-autoplan writes on every run, its attempts.jsonl journal, bound review outputs and checkpoint.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 2 commits November 18, 2026 15:58
…codes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit 2767d22 into main Nov 18, 2026
81 checks passed
@garrytan garrytan mentioned this pull request Nov 19, 2026
capy-ai Bot added a commit that referenced this pull request Nov 19, 2026
- `gstack-artifact validate` no longer passes a run.json that gstack-autoplan never wrote. (#4494)
- /ship's changed-number check no longer stops on tests that only share a word with the change (#4478). (#4496)
- On Capy, /ship no longer edits or comments on pull requests as you. (#4518)
- The destructive-command check stops asking about scratch deletes and catches the real damage it used to miss. (#4513)
- The destructive-command check reads commands more carefully, closing gaps found after the last release. (#4525)
- The Codex readiness check and the second-model review now pick the same model for a custom Codex provider (GST-3089). (#4491)
- gstack's Codex hooks work when your Codex folder's path has a `(`, `&` or other shell character in it. (#4528)
- Uninstalling gstack keeps your own Codex hooks trusted. (#4528)
- Codex hook setup matches Codex more closely. (#4528)
- On Windows, setup copies the Claude Code hook code that gstack's Codex hooks import (#4528)
- /review's checklist on Codex reads shell commands more carefully. (#4528)
- On Codex, gstack now suggests the right skill for a prompt and stops destructive commands before they run. (#4512)
- On Codex, /review is now held once when it tries to finish without its adversarial pass or its report, as it already is on Claude Code. (#4522)
- On Codex, gstack's skill suggestion now names the skill the way Codex lists it. (#4523)
- On Codex, gstack's suggested prompts and docs now name skills the way Codex lists them. (#4524)
- /review on Codex with gpt-6.1-sol is no longer held back for a second-model pass it already ran. (#4526)
- On Codex, office-hours, plan-design-review and gstack-upgrade work again when you name them. (#4510)
- gstack knows when Codex is running unattended. (#4510)
- Codex reviewers start with a clean context. (#4510)
- Docs no longer say Codex has no hook support. (#4510)
- A command whose output was cut off no longer counts as a finished read. (#4492)
- A folder marked as published data can no longer hide source code or agent files from /ship's review. (#4516)
- Correction to the on-by-default helper hooks entry: the hook still routes only to /review, /qa and /verify, so skills you've installed yourself aren't suggested yet. With gstack's skills it routes 14 of 60 real bug reports (was 17), and long prompts are no longer skipped. (#4505)
- /sync-gbrain's read check now works when the first page in your code source is over 64 KB. (#4497)
- Free-suite CI no longer fails before any test runs when Ubuntu's package mirror is slow. (#4530)
- Syncing a second repo no longer makes the first one ask for /sync-gbrain again. (#4532)
- gstack's helper hooks are now on by default in Claude Code, OpenClaw and Hermes, and the prompt-routing hook fires on real bug reports. (#4503)
- `gstack-pregate clear --receipt` accepts a remote gate run from before a merged release when the release only regenerated a declared release output. (#4483)
- The second-model review now keeps the reviewer's own result, and reports a used-up budget as a budget stop. (#4515)
- When the reviewer wrote its review to the result file it was asked for, Codex then replaced that file with its short closing message ("Done, I wrote the review"), so the review was thrown away and reported as missing its markers. (#4515)
- When the provider refuses the call with HTTP 402 Payment Required (usually a metering proxy that hit its allowance, or an account out of credits or quota), gstack used to record only "exit 1". (#4515)
- The pre-commit /verify check now fails a check that times out, and its checks share one time budget (GST-3201). (#4527)
- /ship's review collector no longer reads a reviewer's own scope list or list of passed checks as findings (GST-3201). (#4527)
- Merge-queue stack tests fail when git cannot list a PR's added test files (GST-3201). (#4527)
- /ship can now review results PRs that publish raw session data. (#4511)
- /ship's review pass works when the checkout you ship isn't the project's default checkout. (#4507)
- A second review round no longer reports done when nothing was reviewed. (#4520)
- The prompt-routing hook now suggests skills you install yourself, and no longer sends ordinary bug reports to /qa (GST-3123). (#4506)
- gstack's script runner and merge-queue checks now also refuse cut-off output. (#4502)
- A failed second-model review reports the cause that actually stopped it, and the runner never deletes its own prompt. (#4519)
- `gstack-ship-queue ready` now fills in the trial merges and the second-model trail read instead of printing lines to fill. (#4529)
- Five small ship-tool fixes. (#4529)
- A ruling the owner gives while a `/ship` run is open can be relayed. (#4529)
- /ship no longer kills a working test run as stalled when the test command starts its workers in their own session. (#4533)
- /ship no longer stops a review because the state-machine reader started its reply the way its instructions say. (#4504)
- When Claude Code runs /verify right before a commit, it now gets a quick check instead of the full fix loop (GST-3150). (#4508)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant