Skip to content

Let pull requests that only change human-facing docs skip the test suites (GST-1951) - #4178

Merged
capy-ai[bot] merged 6 commits into
mainfrom
gst-1951-docs-only-prs
Nov 9, 2026
Merged

capy-ai[bot] merged 6 commits into
mainfrom
gst-1951-docs-only-prs

Conversation

@garrytan

@garrytan garrytan commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

A pull request that only changes human-facing docs now skips the test suites and runs a quick link and docs check instead, so docs fixes merge in minutes.

What changed

  • One classifier. lib/docs-only.ts (CLI bin/gstack-docs-only) decides whether every changed path is human docs. The allowlist is root README.md, docs/**/*.md and changelog.d/*.md. A denylist keeps agent instructions, policy docs, templates, sections, pinned results, preregistrations and generated outputs out. Any doc that a script, test, workflow or skill names by path stays out too. Renames check both sides. Any git error means "not docs-only".
  • CI. A docs-scope job in free-tests.yml and windows-smoke.yml runs the base branch's classifier on the merge commit. Suite jobs skip only when docs-scope succeeded and said docs-only. A new docs-lane job checks links on the lines the PR added and runs the tests that read docs by directory. The required free-tests check still always runs, passes a docs-only PR only after docs-lane passed, and records no tested tree.
  • windows-smoke always reports success. The job always runs and skips its steps on a docs-only PR, because gstack's land counts only a check that concluded success.
  • Pregate, CI carry and /ship. On a docs-only change the lanes check runs the same docs lane and owes no remote gate. A docs-only CI run carries as not_owed=docs-only when the local classifier agrees.
  • Land. land.docs_only_skip can drop the merged-tree test list for a docs-only pin. Land keeps every check when a non-docs file on the base tip names a changed doc. The key isn't turned on in .gstack/ship-policy.json yet, because older gstack builds refuse it as unknown.

Why

Garry asked for this on 2026-11-08: "For docs changes pls stop doing full CI and merge without tests. If there are really only docs." Docs-only PRs spent 45-51 Free Tests runner-minutes and 6-8 minutes of wall time each on suites their changes can't affect (runs 42417651501, 42244137054). In the 300 PRs merged 2026-11-01..11-08, 2 qualify (#4144, #4081). The saving inside gstack is small. The bigger value is that every crew's /ship and pregate get the class by default.

Blast radius

Every pull request now runs one extra docs-scope job in free-tests.yml and windows-smoke.yml. Code pull requests run the same suites as before. If the classifier crashes or errors, every suite runs. Pushes to main always run the full suites. Land behavior is unchanged until main's policy turns on land.docs_only_skip.

Pre-Landing Review

Pre-Landing Review: 14 issues — 0 auto-fixed, 9 asked (9 fixed, 0 skipped), 0 deferred, 5 noted
Gate edits: 30 listed, 20 read, 2 findings — 2 kept (reasons below), 0 restored, 0 open

  • RH-15? .github/workflows/free-tests.yml docs-scope/docs-lane hunk: kept, it is the feature (suites skip only on a successful docs-only verdict).
  • RH-15? second tagged workflow hunk: kept for the same reason. The contract tests that pin the workflow shape were updated on purpose and reviewed in both passes.

The full R51 panel ran: specialists (testing, security, state machines), red team, native adversarial, a cross-family outside voice (GPT-6.1 Sol) and the targeted gate reader. Pass 1 ran on c06b33a. Pass 2 re-checked the fixes on c06b33a..f285d09.

Fixed:

  • Land skipped the merged-tree tests when the base tip had gained a reader of a changed doc. It now keeps every check in that case.
  • gstack builds older than this change reject land.docs_only_skip as an unknown key, so the key is held out of main's policy.
  • Two tests that read docs by directory (no-stale-gstack-brain-refs, codex-web-search-flag) are now registered in DOCS_READER_TESTS.
  • The docs lane now sets a git identity for the reader tests that commit fixtures.
  • The link check now covers untracked docs and docs with quoted or spaced names, and it reports an undecodable anchor instead of crashing.
  • windows-smoke no longer skips at job level, which would have failed every docs-only land.

Noted (below the blocking floor; no test, production or data risk today):

  • verifyRun's docs-only branch doesn't confirm that docs-lane itself succeeded, or that the run's merge tree matches the head.
  • The PR's own ./.github/actions/docs-scope action file controls the GitHub check result. gstack's carry still reclassifies locally.
  • A succeeded docs-scope gate also accepts a skip caused by reuse. It is refused later as artifact_missing.
  • The docs-lane link step has no pipefail, so a git diff failure there passes.
  • The land guard catches readers that name a doc by its literal path, not directory scanners added on the base. This is dormant while the policy key is held.

Test Coverage

Coverage: 68% value-weighted (68% including 0 weakly covered paths), 7 gaps. 0 tests added by the audit (audit only). Gaps: the docs-only-lane reader-test branches, the run_changed and api_error paths in the verifyRun docs-only branch, the run-check docsOnly closure, the two land-direct call sites, and a real-action E2E of the docs-scope composite. The check-doc-links CLI gap has since been closed (test added in a91b6c9).

Exploratory QA

Functional CLI probes of gstack-docs-only on an isolated clone: 7 probes, verdict pass. A referenced doc, an agent-instruction name, an executable doc, a push event and this PR's own diff all refuse. An unreferenced new doc is docs-only and selects the 6 reader tests, and a pull_request run writes docs_only=true. Limits: the workflow-level behavior is covered by YAML contract tests and CI only.

Evals

Not run. The selector fell back to all 113 behavioral probes because .github/actions/docs-scope/action.yml is an unknown dependency. That is far over the $10 per-PR cap, and no paid case's reference closure reaches the classifier.

TODOS

No TODO items completed in this PR.

Documentation

Status: updated. One factual anchor fix; no blockers.

Scope: base b615592..f285d09 on gst-1951-docs-only-prs (5 commits, 32 files) plus the unstaged test/docs-only.test.ts. Audited the docs-only PR feature (GST-1951/R81) against lib/docs-only.ts, bin/gstack-docs-only, lib/land-docs-only.ts, lib/pregate/docs-only-lane.ts, lib/ci-carry.ts and the free-tests, windows-smoke and docs-scope workflows.

Documentation health:
docs/pregate.md Updated (moved the flaky-exception anchor back to the Owner-granted flaky exception heading; the new Docs-only section had taken it over)
docs/ship-policy.md Current (land.docs_only_skip row matches land-merged-checks validation)
docs/troubleshooting.md Current (docs-only-error entry matches the CLI's error line and exit 2)
capy/skill/gstack/factory.md Current
ship/sections/tests.md(.tmpl) Current (template and generated copy agree; the not-owed wording matches ci-carry)
changelog.d fragment Current (matches the classifier's allowlist, denylist and fail-closed rules)
CHANGELOG.md / TODOS.md Skipped (ship-owned mode)

Documentation coverage:
gstack-docs-only CLI / docs lane reference: docs/pregate.md#docs-only; how-to: partial (CI reuse recipe); tutorial: none; explanation: docs/pregate.md
land.docs_only_skip reference: docs/ship-policy.md and docs/pregate.md; no how-to

Debt: common gaps only (no tutorial). The docs-only feature is reachable from README only through docs/pregate.md.

Diagram drift: none found.

Test plan

  • bun run test (CI carry): green, run 42544881419, 36641 tests at tree 2dea7989ae5d
  • Focused: bun test test/docs-only.test.ts test/docs-only-carry.test.ts test/docs-only-workflow.test.ts test/ci-reuse-workflow-wiring.test.ts test/cso-distribution.test.ts test/windows-native-workflows.test.ts test/windows-path-scope-workflow.test.ts test/autoplan-residuals.test.ts: pass
  • bun run typecheck: pass; actionlint with shellcheck on both workflows: pass
Agent notes
  • Plan: autoplan-20261108215909, gate_rev=1, approved by Garry ("do it", 2026-11-08). The freeze exception was granted the same day ("Hurry it up").
  • Held land key. land.docs_only_skip is not in .gstack/ship-policy.json. To turn it on, after the first release that contains this change, add "docs_only_skip": ["bun run scripts/test-free-shards.ts --files-from test/land-merged-checks.txt --wall-timeout 600"] under land and raise min_gstack_version to that release in the same PR. Until then land runs every merged check on docs-only PRs.
  • Review gap. The docs-audit anchor move in docs/pregate.md and the check-doc-links CLI exit-code test (both in a91b6c9) landed after pass 2's readers read tree f60b2b. The review-log rows were persisted at tree 2dea79, which includes them, so those two edits got no separate reviewer read.
  • Ledger note. Pass 1's r1 ledger reads dirty because a red-team brief was rendered with an empty findings list and never sent. The sent re-render matches, and pass 2's ledger verifies clean.
  • Deviation from the plan: evals.yml and windows-scoped-tests.yml are not gated (findings F13, F16).
  • lib/land-docs-only.ts is its own module to avoid an import cycle (ship-policy -> land-merged-checks).
  • Existing broken links in docs/ (107 at b615592) aren't fixed here. The link check looks only at lines a PR adds.
  • Sibling trial merges (at 0b20af6): Run checks for files that still name deleted tests (#3536) #4158 merges clean. fix(ship-queue): keep land's merged checks out of the checkout's git config and hooks (GST-1747) #4119 already conflicts with main, with two adjacent lines in lib/ship-queue-land-direct.ts.
  • REVIEW_LOOP: passes=2 blocking_found=11 deferred=0 noted=5 fixed=9 scope=frozen unrecorded=5 ledgers_read=4 dispatches=8
  • Gate edits: 30 listed, 20 read, 2 findings — 2 kept (reasons above), 0 restored, 0 open

Approvals

No approvals recorded for branch gst-1951-docs-only-prs.
Plan: approved (record r-4bbb070b2d75)
Plan check: approved (record r-4bbb070b2d75)

Refs GST-1951

{
  "schema_version": 1,
  "pr": 4178,
  "head": "a91b6c9d62398aabf50d13a3a8f5b1bb6f3edc00",
  "version": "1.98.0.0",
  "base": "main",
  "gate": {
    "test": {
      "pass": 1648,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "test",
      "run": 42544881419,
      "attempt": 1,
      "tree": "2dea7989ae5de957ab62a1aa9554177dd0e917d1"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "2dea7989ae5de957ab62a1aa9554177dd0e917d1",
  "product_patch_id": "385af1b61bfc6db23c5f2ad9b1f0b16eb99dbade",
  "gstack_version": "1.98.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 702,
    "rules": "66e5174161eecc2c",
    "base": "b6155921a564ba9ec2470fd2cb3894c05e2c2d13",
    "registry": "b6155921a564ba9ec2470fd2cb3894c05e2c2d13",
    "gate_scan": {
      "tagged": 2,
      "head": "a91b6c9d62398aabf50d13a3a8f5b1bb6f3edc00"
    },
    "reasons": [
      "gate-class",
      "risk:security",
      "runtime=702"
    ],
    "risk": [
      "security"
    ],
    "mode": "panel"
  },
  "pregate": "requires-remote cleared @2dea7989ae5d (8 checks, current for a91b6c9d6239)",
  "requires_remote": [
    "free-tests.yml/cso-macos-launcher: https://github.com/garrytan/gstack/actions/runs/42544881419/attempts/1",
    "free-tests.yml/macos-named-regressions: https://github.com/garrytan/gstack/actions/runs/42544881419/attempts/1",
    "free-tests.yml/cso-windows-launcher: https://github.com/garrytan/gstack/actions/runs/42544881419/attempts/1",
    "windows-smoke.yml/windows-smoke: https://github.com/garrytan/gstack/actions/runs/42544881240/attempts/1",
    "pregate.json/remote_gate: remote-gate tree=2dea7989ae5d tests=36641 skipped=105 runs=42544881419/1 profiles=ci-reuse/free-tests heads=a91b6c9d6239 verified=2026-11-09T01:26:15.609Z"
  ],
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "docs/, README.md, changelog.d/ (docs-only PR feature against its code and workflows)"
  },
  "review_loop": {
    "passes": 2,
    "blocking_found": 11,
    "deferred": 0,
    "noted": 5,
    "fixed": 9,
    "scope": "frozen",
    "rounds": 2,
    "unrecorded": 5,
    "ledgers_read": 4,
    "dispatches": {
      "specialist": 6,
      "red-team": 2
    }
  },
  "review_targeted": {
    "classes": [
      "gate"
    ],
    "rules": "8811fd300bf305f0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "385af1b61bfc6db23c5f2ad9b1f0b16eb99dbade",
    "status": "done",
    "findings": 1,
    "raised_by": "targeted"
  }
}

…ites (GST-1951)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
garrytan and others added 4 commits November 9, 2026 00:48
…ate the workflow contract tests

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… doc, hold the land policy key until a release reads it, and register two more docs reader tests

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…nd report an undecodable anchor instead of crashing

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…k checker's exit codes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit e077053 into main Nov 9, 2026
71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant