Skip to content

v1.94.6.0 feat(hooks): opt-in skill routing and contract Stop hooks with gstack-skill-contract (GST-1088 lane A) - #3912

Merged
capy-ai[bot] merged 10 commits into
mainfrom
capy/gst-1088-lane-a
Nov 3, 2026
Merged

capy-ai[bot] merged 10 commits into
mainfrom
capy/gst-1088-lane-a

Conversation

@garrytan

@garrytan garrytan commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 3912,
  "head": "5c7d678bb98cc8bf7c0996045b2b5ebc4e1b1419",
  "version": "1.93.34.0",
  "base": "origin/main",
  "spend_usd": 6.55,
  "session_kind": "unattended",
  "run": "jam_01M6FGN8NNX1Y5JK5GZAMD8J4F",
  "thread": "GSTA-256",
  "head_tree": "084ea6f2469f2dc23cea8be86f31b3c7ef03d35c",
  "product_patch_id": "bd3e324e1451b8727332fcb6f7bf76f74a244a5a",
  "gstack_version": "1.93.34.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 1407,
    "rules": "b2272efd321ce7cb",
    "base": "6f1dfd5b7cd0507d29e62d09a31e77f7e787e947",
    "registry": "6f1dfd5b7cd0507d29e62d09a31e77f7e787e947",
    "gate_scan": {
      "tagged": 2,
      "head": "5c7d678bb98cc8bf7c0996045b2b5ebc4e1b1419"
    },
    "reasons": [
      "gate-class",
      "runtime=1407"
    ],
    "risk": []
  },
  "pregate": "requires-remote cleared @084ea6f2469f (1 checks, current for 5c7d678bb98c)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/41572876840/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/41572876840/attempts/1",
    "windows-setup-e2e.yml/windows-setup: https://github.com/garrytan/gstack/actions/runs/41572876863/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/41572876858/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=084ea6f2469f tests=20430"
  ],
  "docs": {
    "status": "updated",
    "scope": "docs/skill-routing-and-contracts.md"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 2,
    "deferred": 0,
    "noted": 9,
    "fixed": 2
  },
  "review_targeted": {
    "classes": [
      "gate"
    ],
    "rules": "c82addff6fb60fa0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "bd3e324e1451b8727332fcb6f7bf76f74a244a5a",
    "status": "done",
    "findings": 0,
    "raised_by": "targeted"
  }
}

What this does

Two opt-in Claude Code hooks so a model actually runs gstack's skills (GST-1088, lane A of plan-v2). Both switches default off, so this PR changes no behavior when it merges.

  • Routing hook (hosts/claude/hooks/skill-route-hook, UserPromptSubmit, skill_routing). Scores the prompt against the installed skills' triggers: frontmatter (no model call, no network) and, at or above the 0.66 threshold, adds one additionalContext line that names its source and its off switch and asks the model to say in one sentence that gstack routed it. Slash-command prompts, prompts over 8,000 characters and every error produce no output; it always exits 0 (never 2) under a 1.5 s internal deadline. Trace and per-session failure counter in $GSTACK_STATE_ROOT/hooks/skill-route.log, never the prompt text, never stderr.
  • Contract Stop hook (hosts/claude/hooks/skill-contract-stop-hook, Stop, skill_contracts). Checks the most recently loaded skill that declares a contract: block (schema v1 frozen in plan-v2: report_labels, check_after_last_edit, selected_specialists_dispatched, agent_dispatch) and blocks once per skill run with the gate rev 2 E1 text, which never names the off switch. It checks only a final turn (a STATUS line, a headless or unattended session, or a turn that does not end on a question), allows at once on stop_hook_active, and allows while Agent dispatches are still running. Every read, write, parse or timeout error fails open with one stderr line ending (see docs/skill-routing-and-contracts.md#state-error); an unknown transcript shape also bumps a canary counter. Records are written atomically per Claude Code session id before any block.
  • gstack-skill-contract check|report|waive: check is a pure function of a transcript JSONL (evals can reuse it); waive is stateless (G1) and check reads waivers from the waive Bash calls in the transcript, counting them as misses for follow; report lists misses and re-checks a blocked run against its transcript. Stable exit codes 0–6, each with a docs anchor.
  • gstack-skill-route --explain "<prompt>": candidates, scores, threshold, route and the exact hint (a dry run, works with the switch off).
  • setup: --[no-]skill-routing and --[no-]skill-contracts, mirroring the timeline_stop_hook gate: flag > env (GSTACK_SKILL_ROUTING / GSTACK_SKILL_CONTRACTS) > config (skill_routing / skill_contracts, on|off) > default off; a flag persists, a typo warns and is not persisted, off and --no-team remove the registration, and the first registration prints one line. Two KNOWN_HOOKS rows in bin/gstack-settings-hook.
  • Docs: docs/skill-routing-and-contracts.md (precedence, dry run, schema with a worked example, host support table, check/report/waive, an anchor per error code).

Field names match lane B (#3909): marker is literal text where a whole-word N stands for a number (Dispatching N specialists:), and a "Small diff ... specialists skipped." line counts as zero specialists selected, a pass.

Security note (install surface)

  • setup gains two flags and a gate function, and bin/gstack-settings-hook gains two identity rows. With a switch on, setup writes a hook entry into ~/.claude/settings.json through the existing hardened settings-hook (lock, backup-on-change, atomic write, fail-closed on a corrupt file). With both off (the default) setup only calls remove-source, which writes nothing when no entry exists.
  • Neither hook makes a network call, spawns a model, or reads environment secrets. The Stop hook spawns bin/gstack-session-kind (1 s cap) only when a turn ends on a question. Both read the transcript and SKILL.md files the host already has; logs and records are mode 0600 under the state root and never hold prompt text.
  • The routing line is agent-facing and names its off switch (plan d12, as briefed); the Stop block text does not (E1).

Scorer precision/recall on Run D (d7) and hint pilot (d6)

d7, deterministic scorer, no model. Inputs: the 37 Run D tasks (gstack-evals 8cd4973 quick/laneb/tasks/final.json, less crack-7z-hash, adaptive-rejection-sampler and largest-eigenval), instructions from terminal-bench-2 69671fb read as text. Fit stratum (should route to /investigate): custom-memory-heap-crash, overfull-hbox, fix-code-vulnerability, fix-git, db-wal-recovery. The other 32 are decoys. Triggers: lane B #3909 at b772dc6 and again at its current head cc1539c (identical rows: cc1539c drops two /review triggers that never fired on these tasks), and main's (54606c5) as the before.

Triggers, threshold Fit routed to /investigate /investigate precision Decoys routed to any skill
main, 0.66 1/5 1/1 2/32
lane B, 0.5 3/5 3/16 26/32
lane B, 0.66 (shipped) 2/5 2/4 4/32
lane B, 0.75 and 1.0 2/5 2/4 4/32
lane B, 0.66, every installed skill routable (42ee037) 2/5 2/4 17/32

The last row is why 93b17ed limits routing to the four discovery targets in plan decision 4 (/investigate, /review, /qa, /verify): with all 63 skills routable, 17 of 32 decoys got a hint for some unrelated skill. At 0.66 the four decoy routes are extract-elf and constraints-scheduling to /investigate, and torch-pipeline-parallelism and tune-mjcf to /verify. The three fit misses (overfull-hbox, fix-git, db-wal-recovery) are short task specs that share no /investigate trigger. Recall is lane B's trigger list, not the scorer, and 0.66 stays provisional until GEVA-214 freezes it.

d6, hint pilot. 36 claude -p trials, Opus 5.5 (claude-opus-5-5, Claude Code 2.1.317, its own claude_code system prompt), lane B's skills installed, a two-file Node repo with one failing test, three bug-fix prompts × 3 reps per arm, --max-turns 2. Opened = the first tool call is Skill: investigate. Run twice: on lane B b772dc6 ($3.03) and on its head cc1539c ($3.23), with the same result both times; lane spend $6.41 with the smoke trial.

Arm Opened /investigate
hook off (control) 0/9 and 0/9 (first call Bash every time)
name (default) 9/9 and 9/9
name + first step 9/9 and 9/9
name + contract 9/9 and 9/9

The hint moves Opus from 0/9 to 9/9, and it said "gstack routed this to /investigate" as asked. The three wordings tie at the ceiling, so name, the shortest, stays the default. Why null for the wording comparison: exposure was confirmed (the hook logged a route in every hinted trial and /investigate was in every session's skill list), but these prompts are easy and saturated all three arms, so the pilot can't rank the wordings. Verdict: KEEP the hint; FIX-METHOD for the wording comparison (harder, ambiguous prompts in GEVA-214's run). The CLAUDE.md-block fallback isn't triggered, because the hint did move Opus.

Review (gstack-review-plan, full depth, class gate)

  • Pass 1 at 93b17ed: five readers (batched testing/security/api-contract/state-machines, red team, adversarial, targeted gate reader on Opus 5.5; outside voice on GPT-6.1 Sol). The outside voice found two blocking bugs: one background agent's completion notice cleared every pending agent, so the Stop hook could block while another specialist was still running, and report lost a fixed run once the skill was loaded again. Both are fixed in 798045b, along with six informational fixes: stop_hook_active Stops now record met_after_block/miss (never block), the payload's last_assistant_message covers a transcript that lags one message, a met run stays met, only a SKILL.md read under a skills/ directory counts as a load, specialist selection lines come only from assistant text or Bash results with plain names (closes a prompt-injection path into the block text), and check commands match by the program they run.
  • Pass 2 on 93b17ed..798045b: three readers, no blocking findings. dd796cd fixes two notes below the floor: test runs behind wrappers (bundle exec rspec, uv run pytest, docker compose run app pytest) count again, and met_after_block is final too. The review plan stops at the round-2 cap.
  • Open notes, not fixed: (1) single words of five or more letters still route ("add a debug flag" goes to /investigate); changing that would move the d7 numbers above, so it waits for GEVA-214. (2) A run whose first final turn prints its report before any edit is recorded met, and later edits in that run are not checked; that's a product call. (3) remove-source rewrites a settings.json that holds empty hook arrays on every ./setup (core bin/gstack-settings-hook behavior, out of this lane's scope). (4) npm install, node --version and make clean still count as check commands, and one Agent prompt naming two specialists satisfies both; both err toward allowing the stop. (5) When another Stop hook continued the turn first and our contract is unmet, nothing is recorded for that stop. (6) Edits made through Bash (sed -i, heredocs) don't count as edits for check_after_last_edit.

e318060, c19a685, e49c9be, 5c7d678 (lane B and lane C requests): Dispatching N specialists: reads N as a number, and N = 0 is met with no dispatches; parenthesized notes are not names (Dispatching 1 specialists: State Machines (small diff)). agent_dispatch takes an optional marker and prefers it (Dispatching adversarial pass: native followed by an Agent call), keeping match as the fallback. An early exit meets the contract: a final STATUS: BLOCKED (for /investigate, Status: BLOCKED in the DEBUG REPORT) or a final message line starting "Nothing to review" (/review's early exit). Also from e318060: the checker's default marker strings now live in one table, DEFAULT_MATCHERS in lib/skill-contract.ts, so lane C's fixed markers are a one-line edit; Dispatching 0 specialists: (small diff) counts as zero selected. The ledger-backed targeted reader ran after each of these heads; its informational notes are folded into the list above, and on the final head 5c7d678 it reported no findings.

Tests

test/skill-route.test.ts, test/skill-contract.test.ts, test/skill-contract-stop-hook.test.ts, test/setup-skill-hooks-gate.test.ts (fixture skills carry the contract blocks; no skill template is touched): unwritable state root, corrupt record, stop_hook_active, a multi-turn interactive fixture that is never blocked, the J1 /investigate fixture that is blocked, Agent readers still running, the pinned block message (test/fixtures/skill-contract/block-message.txt), routing error cases (malformed triggers, unreadable config, bad JSON, over-long prompt, stdin over cap), hook output schema and settings.json entry schema, and both off switches.

Open in Capy

…ll-contract and gstack-skill-route (GST-1088 lane A)

Both switches default off. UserPromptSubmit routing hook scores prompts against installed triggers; Stop hook blocks once when a contract skill ends without its outputs, failing open on every state error. setup gains --[no-]skill-routing and --[no-]skill-contracts mirroring the timeline gate.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 9 commits November 3, 2026 02:22
…t-of-test ratchet)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…-contract fixture as free (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…op_hook_active, read last_assistant_message, per-agent completion, met runs stay met, installed-only SKILL.md loads, program-based check commands, safe specialist names, report rechecks the saved run (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…t, the shell test builtin does not; met_after_block is final (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…0 selects no specialists (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…line, parenthesized notes are not specialist names (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… the contract (GST-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…T-1088 lane A)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@github-actions github-actions Bot changed the title feat(hooks): opt-in skill routing and contract Stop hooks with gstack-skill-contract (GST-1088 lane A) v1.94.6.0 feat(hooks): opt-in skill routing and contract Stop hooks with gstack-skill-contract (GST-1088 lane A) Nov 3, 2026
@capy-ai
capy-ai Bot merged commit 6a370bd into main Nov 3, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant