Skip to content

feat: retire /investigate after a no-gain eval (GST-1717) - #4110

Merged
capy-ai[bot] merged 15 commits into
mainfrom
capy/gst-1717-retire-investigate
Nov 7, 2026
Merged

capy-ai[bot] merged 15 commits into
mainfrom
capy/gst-1717-retire-investigate

Conversation

@garrytan

@garrytan garrytan commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 4110,
  "head": "754d2e68e4e643af99ba9a9277239b4f7c521c6f",
  "version": "1.97.0.0",
  "base": "main",
  "gate": {
    "test": {
      "pass": 1623,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "test",
      "run": 42347452222,
      "attempt": 1,
      "tree": "33ae5a841a6ec1dd448f9c56dbb1f259907aca0c"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "33ae5a841a6ec1dd448f9c56dbb1f259907aca0c",
  "product_patch_id": "3b7d408c332323ac1af7b83f6845c61bebab1116",
  "gstack_version": "1.97.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 1686,
    "rules": "70895942bb8a8ee2",
    "base": "94392a61ecba8ad674b750b1e93bcc9eeb14bd84",
    "registry": "94392a61ecba8ad674b750b1e93bcc9eeb14bd84",
    "gate_scan": {
      "tagged": 17,
      "head": "754d2e68e4e643af99ba9a9277239b4f7c521c6f"
    },
    "reasons": [
      "gate-class",
      "risk:security",
      "runtime=1686"
    ],
    "risk": [
      "security"
    ],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @33ae5a841a6e (8 checks, current for 754d2e68e4e6)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/42347452222/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/42347452222/attempts/1",
    "windows-setup-e2e.yml/windows-setup: https://github.com/garrytan/gstack/actions/runs/42347452549/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/42347452166/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=33ae5a841a6e tests=35963 skipped=105 runs=42347452222/1 profiles=ci-reuse/free-tests heads=754d2e68e4e6 verified=2026-11-07T06:41:56.757Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "current",
    "scope": "AGENTS.md, README.md, CLAUDE.md, CONTRIBUTING.md and docs/ (base 3742ee95 to e1ef7d80, re-audit of the main merge 754d2e68)"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 2,
    "deferred": 1,
    "noted": 8,
    "fixed": 4,
    "scope": "frozen",
    "fix_checks": 4,
    "rounds": 5,
    "dispatches": {
      "specialist": 1,
      "red-team": 1
    }
  },
  "review_loop_exception": {
    "accepted_by": "GStack Minder",
    "date": "2026-11-07",
    "via": "a-f2a51169043f",
    "reason": "merge-resolution hunks after the rule-13 merge of main: two test files",
    "fix_check": {
      "head": "754d2e68e4e643af99ba9a9277239b4f7c521c6f",
      "findings": 2,
      "files": [
        "test/eng-finding-retry-budget.test.ts",
        "test/qa-supervision-selection.test.ts"
      ]
    }
  },
  "review_targeted": {
    "classes": [
      "gate"
    ],
    "rules": "30e6bce187be0d59",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "3b7d408c332323ac1af7b83f6845c61bebab1116",
    "status": "done",
    "findings": 2,
    "raised_by": "targeted"
  }
}

What changed

/investigate is retired. It no longer installs on any host, and the next ./setup, relink or upgrade removes an existing gstack-owned investigate / gstack-investigate entry, even though the skill's source is gone. A user's own skill under that name is left alone. Bug reports now get fixed directly. The agent makes one read-only check of the cause, and when it is still unclear, a focused test that fails on the bug before any edit, then the root-cause fix and a rerun.

Why

Garry approved the cut on 2026-11-07 ("do it" on A1, R48 CUT). A preregistered eval found no gain. On 18 real bug-fix tasks that plain Opus 5.5 solves about half the time, 20 trials each, Opus told to use /investigate passed 50.0% of trials against 47.2% with no skill (+2.8 points, 95% CI -5.8 to +11.4, MDD 10), and a passing fix cost about 43% more (GEVA-380 R3c).

Blast radius

The change touches the router, the CLAUDE.md routing block gstack offers, /ship's test-failure triage, /ios-fix, /learn, /verify, setup, relink, the install registry and the docs. It is safe because the retire path is #4108's (RETIRED_SKILLS, GSTACK_RETIRED_SKILLS), plus one relink loop for a retired skill whose source no longer exists, proved by test/retired-skills.test.ts (two of its tests fail against main's relink and registry). Evidence rung: script ran.

Approvals

Approved: yes (record r-6c5eb42b9094). The full approval record stays private: this repository is public or its visibility is unknown.

Summary

  • Retire path. scripts/discover-skills.ts and bin/gstack-install-registry.sh list investigate as retired, so config can't re-enable it. bin/gstack-relink now cleans <skills>/investigate and <skills>/gstack-investigate for every retired skill, because a deleted source never reaches the main loop. The OpenClaw and QM editions drop their investigate skills.
  • The opt-in /investigate gate. Its hooks, lib/investigate-gate.ts and its switch in gstack-config, gstack-skill-gates and gstack-skill-route are gone. ./setup removes the gate's four settings entries on every run; the KNOWN_HOOKS rows stay so the old entries are still recognized as gstack's.
  • Shared machinery kept. skill-contract-stop-hook and lib/skill-contract.ts stay for /review's contract; lib/skill-gates.ts stays for the /plan-ceo-review and /autoplan gates. lib/ceo-review-gate.ts carries its own git helpers instead of importing them from the deleted gate. ROUTED_SKILLS is now review, qa and verify.
  • No removed skill recommended. The router, the preamble triage, bin/gstack-skill-start, CLAUDE.md, /ship (triage and the Blast radius evidence-ladder rungs, now inline), /ios-fix, /learn, /verify, review/checklist.md, the gbrain and question registries, the doctor components and the docs all point at fixing directly. docs/skills.md gets a /investigate (cut) note with the evidence link.
  • PR selector. A diff that deletes a prompt file no longer throws PR profile requires full validation for a file with no prompt left (scripts/test-pr-profile.ts; the new test fails without the fix). careful/bin/hook-extract.sh is free-only now that its only paid consumers retired.

Paid cases

Case Tier Fate Why
investigate-owned-completion gate retired tested /investigate's own completion prose; test file deleted
investigate-owned-abort gate retired same, abort ending
investigate-owned-ending-error gate retired same, error ending
carve-section-loading-investigate periodic retired read /investigate's carved sections; CARVE_GUARDS.investigate removed
codex-sol-scope-termination periodic retired installed the full generated investigate skill as its vehicle; its test and fixture are deleted
router-failure-contradicted-cause gate retired its contract was "route to /investigate before any edit"
router-failure-triage-unclear gate retired same
router-failure-verified-cause gate kept, changed the direct-fix path: no skill route, suite green, 1 to 4 reads. Run once: PASS, $0.09
router-failure-triage-resolves gate kept, changed same. Run once: PASS, $0.07
journey-debug periodic kept, changed now expects a direct answer with no skill call. Run once: PASS, $0.05
model-panel-investigate periodic kept a bug-fix task named "investigate"; it never loaded the skill

None of the retired cases was quarantined, so the CASE_QUARANTINE exit rule does not apply; each is removed from E2E_TOUCHFILES, E2E_TIERS, E2E_KINDS, the host map, the exercised-command map and the duration files, with this table as its record. The retired fallback router cases leave the unclear-cause path ("failing test first") with no paid negative control.

Test Coverage

CI at 754d2e6 is green: 36 checks passed, 0 failed, 0 pending. The free suite ran 1623 test files in run 42347452222, carried as CI_CARRY at tree 33ae5a841a6e. The macOS and Windows lanes cleared the pregate's requires-remote obligations. Local focused runs of test/retired-skills.test.ts and the two merge-resolution test files were green. The three changed paid cases ran once each under Claude Code 2.1.292 (run log gst1717-routing) and each passed, for $0.21 total.

Pre-Landing Review

REVIEW_LOOP: passes=1 blocking_found=2 deferred=1 noted=8 fixed=4 scope=frozen fix_checks=4 rounds=5

Both blocking findings were fixed with failing-first tests. They were the prune data-loss case and the per-host backup collision. The final adversarial pass at e1ef7d8 found 3 Low items and the outside voice found none. The two later fix-checks (acdd017 and 754d2e6) came back clean.

Review-loop exception: accepted by GStack Minder 2026-11-07 via a-f2a51169043f: merge-resolution hunks after the rule-13 merge of main: two test files (fix-check at 754d2e6, 2 findings)

Open notes, none blocking:

  • bin/gstack-relink still removes a marked investigate directory when a stale render survives a failed re-render.
  • Windows copy installs can leave an inert investigate/sections/ folder.
  • Backup labels collide for host paths that differ only by / and _.
  • scripts/test-pr-profile.ts:499 checks deletion against the selector's checkout, not the reviewed tree, and exempts a deleted prompt that is still referenced.
  • bin/gstack-relink:269 has dangling sections/ links, docs/CONTRIBUTOR_REFERENCE.md:86 names the old backup path, and test/skill-gates-advisory-bytes.test.ts:23 dropped the plan-ceo-review source scan.
  • review/checklist.md:129 still names /investigate on purpose until GST-1744.

Documentation

Status: current. Audit ship4110-docs-a1 covered base 3742ee9 to e1ef7d8, and re-audit ship4110-docs-a2 covered the main merge in 754d2e6. Every authored doc already describes /investigate as retired, so neither audit made edits. Debt: the generated docs/inventory.md still lists investigate until bin/gstack-inventory regenerates it.

Open in Capy

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 14 commits November 7, 2026 05:13
…GST-1717)

The retired /investigate rule became 'Bugs/errors → fix them directly', which no
longer matches '→ invoke /', so proactive=core leaked it into the offered section.
Match any '- … → ' rule line.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…(GST-1717)

The pin names replay GSTA-224 H1; a reviewer checklist edit merges only after a
passing no-loss replay (crew rule 15). The /investigate flake line stays until a
replayed repin.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
CI run 42340956269 at 83a8a9a: drop fixtures, ledger rows, allowlist entries and
counts for the retired skill and its deleted paid files; record the harness
change as non-behavioral; the test-of-test detector accepts path.join(ROOT, 'bin');
uncovered-prompt fixtures use a tracked prompt, since an untracked path now
reads as a deleted one.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…it (GST-1717)

With the skill's source gone, the customized-file check never ran, so a
banner-carrying local copy of investigate/SKILL.md was deleted instead of moved
to the backup root (red-team review, pass 1).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
review/checklist.md keeps its pinned /investigate line, so the QM edition needs
its replaceText rule back; qm/ stays fresh against the generator.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… pinned checklist line (GST-1717)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… deleting it (GST-1717)

Scoped fix round under grant a-360dd62bd46a (outside-voice blocking finding and
adversarial #1, #4). setup's prune compares a copied host entry with the render
before removing that render, and backs up a copy that differs or whose render is
gone; relink no longer takes the marker's rm -rf for a skill whose source is
gone, so a real SKILL.md is backed up and the user's files stay. setup reports a
failed remove-source of the retired gate hooks on stderr.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…one from disk (GST-1717)

An untracked prompt file on disk still needs prompt validation (targeted review
finding); harness change recorded as non-behavioral.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ST-1744 (GST-1717)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…GST-1717)

Under grant a-c95576b8b1f9 (fix-check#1): every copy host moved its
gstack-investigate/SKILL.md to the same backup path, so a later host's copy
overwrote an earlier host's edits. A sandboxed ./setup --host factory run showed
gen-skill-docs removes the retired render before the prune, so every real copy
is backed up; each host now gets its own folder.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…(GST-1717)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…1717)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ral waiver (GST-1717)

pregate's literals check read the GSTACK_RETIRED_SKILLS change as an old
value still in the tests. plan-eng-review is still retired; investigate was
appended. The fixture names it once, with the waiver.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…t, GStack Minder 2026-11-07)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant