Skip to content

Let Trunk watch the Windows checks, quality gate and eval jobs on pull requests, learning only (GST-1847) - #4148

Merged
capy-ai[bot] merged 4 commits into
mainfrom
capy/trunk-watches-more-pr-checks-gst-1847
Nov 8, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
capy/trunk-watches-more-pr-checks-gst-1847

Conversation

@garrytan

@garrytan garrytan commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner
{
  "schema_version": 1,
  "pr": 4148,
  "head": "cebb98e5d9e2bfe9d43928b0557b8226d30cb99b",
  "version": "1.98.0.0",
  "base": "main",
  "gate": {
    "free": {
      "pass": 36264,
      "fail": 0,
      "skip": 0
    }
  },
  "ci_carried": [
    {
      "lane": "free",
      "run": 42430253538,
      "attempt": 1,
      "tree": "17390d67f5d8e08476a0785e261734835b059c71"
    }
  ],
  "session_kind": "unattended",
  "head_tree": "17390d67f5d8e08476a0785e261734835b059c71",
  "product_patch_id": "6412ce014dd6fefe58956b200983547e5944eb63",
  "gstack_version": "1.98.0.0",
  "review_depth": {
    "depth": "full",
    "class": "gate",
    "runtime_lines": 155,
    "rules": "c6a079fdf812f479",
    "base": "aa46032f2be715cba570af61d6a059a2853fba7d",
    "registry": "aa46032f2be715cba570af61d6a059a2853fba7d",
    "gate_scan": {
      "tagged": 10,
      "head": "cebb98e5d9e2bfe9d43928b0557b8226d30cb99b"
    },
    "reasons": [
      "gate-class",
      "runtime=155"
    ],
    "risk": [],
    "mode": "panel"
  },
  "pregate": "pass @17390d67f5d8 (8 checks, current for cebb98e5d9e2)",
  "issues": {
    "fixes": [],
    "refs": [],
    "flakes": []
  },
  "docs": {
    "status": "updated",
    "scope": "AGENTS.md, CLAUDE.md, CONTRIBUTING.md, README.md and docs/ (142 files) against aa46032f..HEAD; edited docs/TESTING_INTERNALS.md"
  },
  "review_loop": {
    "passes": 1,
    "blocking_found": 0,
    "deferred": 0,
    "noted": 0,
    "fixed": 0,
    "scope": "frozen",
    "rounds": 1,
    "ledgers_read": 4,
    "dispatches": {
      "specialist": 3,
      "red-team": 1
    }
  },
  "review_targeted": {
    "classes": [
      "gate"
    ],
    "rules": "8811fd300bf305f0",
    "brief_sha": "7f92a0d0a8319c3c0e076fb3b1a4689c45120b2be2c14dfc7d4cb03aafa9d6a5",
    "product_patch_id": "6412ce014dd6fefe58956b200983547e5944eb63",
    "status": "done",
    "findings": 0,
    "raised_by": "targeted"
  }
}

Trunk's learning-mode observer now also records would-run/would-skip verdicts for the Windows checks, the quality gate and the eval jobs on pull requests, so the measurement covers the expensive PR checks and not just the free suite. Nothing reads the verdicts, so every check still runs with its own name and result; this serves the README goal of keeping CI fast and cheap without losing signal.

What changed

The testing docs now list all six observed workflows and say which skipped jobs to leave out at the review date

The testing docs now list all six observed workflows and say which skipped jobs to leave out at the review date.

Before/after as text: The testing docs now list all six observed workflows and say which skipped jobs to leave out at the review date
--- Before · docs/TESTING_INTERNALS.md § Trunk Dynamic CI (learning only) at aa46032 · source text
+++ After · docs/TESTING_INTERNALS.md § Trunk Dynamic CI (learning only) on this branch · source text
@@ ⋯ 6 unchanged ⋯ @@
 free-suite shards share one verdict. The verdicts show up in the job summary,
 on the pull request's Dynamic CI tab and at app.trunk.io/dynamic-ci.
 
+Trunk keys verdicts by workflow file, so it answers only for jobs in the
+workflow that ran the action. So since GST-1847, the other costly pull-request
+workflows carry the same observer job, each named "Trunk Dynamic CI (learning
+only, <workflow>)":
+
+| Workflow | Observed jobs |
+|---|---|
+| `free-tests.yml` | `free-suite`, `macos-named-regressions` |
+| `windows-smoke.yml` | `windows-smoke` |
+| `windows-scoped-tests.yml` | `windows-scoped-shard` |
+| `windows-setup-e2e.yml` | `windows-setup`, `held-install-journey` |
+| `quality-gate.yml` | `quality` |
+| `evals.yml` | `build-image`, `eval-slices` |
+
+Everything below holds for every observer.
+
 It never skips anything. No job needs the observer, no condition reads its
 verdicts, and it exports no outputs. Switching Trunk to enforcement in its
+dashboard would still skip nothing, because nothing in its workflow reads
-dashboard would still skip nothing, because nothing in this workflow reads
 the verdicts. Every test runs, and every required check keeps its name and
 its result. Merge evidence never reads Trunk: the `free-tests` aggregate,
 the CI reuse profile, `gstack-ci-carry`, the pregate, the train gate
@@ ⋯ 9 unchanged ⋯ @@
 `gstack-ship-queue ready` counts every check at the head. Rerun it once;
 if it fails again, delete the job. Fork pull requests, pushes and dispatches
 skip the job. A missing token skips only its step, so the job still fetches
+the action. Its only secret is `TRUNK_API_TOKEN` (in free-tests.yml it is
+the only job that sees any secret; evals.yml's other jobs hold their own), so
+it checks out nothing and runs none of the pull request's code. It holds no GITHUB_TOKEN scopes (`permissions: {}`). At the
-the action. It is the only job in the workflow that sees a secret
-(`TRUNK_API_TOKEN`), so it checks out nothing and runs none of the pull
-request's code. It holds no GITHUB_TOKEN scopes (`permissions: {}`). At the
 pinned commit (`src/config.ts`, `src/changed-files.ts`), the action makes one
 network call: a request for verdicts to
 `api.trunk.io/v2/dynamic-ci/generate-plan`, at most 3 attempts with a
 30-second budget each. Telemetry is off. With no checkout and an empty
 `github-token`, the action sends no file list, and Trunk reads the diff itself
 through its GitHub App.
+`test/free-tests-workflow-wiring.test.ts`,
+`test/ci-reuse-workflow-wiring.test.ts` and
+`test/trunk-dynamic-ci-observers.test.ts` lock that shape in place.
-`test/free-tests-workflow-wiring.test.ts` and
-`test/ci-reuse-workflow-wiring.test.ts` lock that shape in place.
 
 **Review date: one week after it merges (2026-11-12 if it merges on
 2026-11-05).** Compare Trunk's would-skip verdicts with what the jobs it
 would have skipped actually found, and look hardest at any would-skip on a run
+where that job failed. Leave out runs where the workflow itself skipped the
+observed job, because Trunk still records a verdict there but there is no
+result to compare it with: `reuse` skipping the Free Tests suite,
+`windows-scope` skipping the scoped-test and setup Windows jobs on a pull
+request that touches no Windows path (`windows-smoke` always runs), and `eval-slices` on pull-request pushes, which run evals.yml
+in its screening-summary mode and plan no slices.
+In free-tests.yml, Trunk's `required-check` vote treats both observed jobs as
+not required, because branch protection names the `free-tests` aggregate and
-where that job failed. Leave out runs where `reuse` skipped the suite: Trunk
-still records a verdict there, but there is no result to compare it with.
-Trunk's `required-check` vote treats both jobs as not
-required, because branch protection names the `free-tests` aggregate and
 not the jobs under it. But `free-tests` needs both jobs and every shard
+receipt, so discount that vote when you judge their would-skip rate. Any skip
+added there later also has to pass the aggregate's receipt check. Turning on
-receipt, so discount that vote when you judge the would-skip rate. Any skip
-added later also has to pass the aggregate's receipt check. Turning on
 skipping is a separate decision for Garry, made after that review.
  • windows-smoke.yml, windows-scoped-tests.yml, windows-setup-e2e.yml, quality-gate.yml and evals.yml each get the learning-only trunk-dynamic-ci job that free-tests.yml has had since ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) #4012. Trunk answers only for jobs in the workflow that ran the action (the action sends GITHUB_WORKFLOW_REF's path and the service enumerates that workflow), so one observer in free-tests.yml can't cover the others.
  • Observed jobs: windows-smoke, windows-scoped-shard, windows-setup + held-install-journey, quality, build-image + eval-slices. All are plain jobs (no reusable-workflow uses: call jobs, which is what gave gbrain WORKFLOW_NOT_RECOGNIZED).
  • Each observer gets a unique check name, Trunk Dynamic CI (learning only, <workflow>), so no two checks at a head share a name.
  • test/trunk-dynamic-ci-observers.test.ts holds all six observers to one shape. The existing workflow tests that pinned exact job lists or "no secrets" for these files now name the observer explicitly (evals-train-workflow, windows-native-workflows, windows-path-scope-workflow).
  • docs/TESTING_INTERNALS.md lists every observed workflow and job, limits the "only job that sees a secret" claim to free-tests.yml, and says which runs to leave out at the review date for each workflow.

Why

Trunk only answered for the free suite, so the review could not say anything about the Windows, quality and eval checks, which cost the most per pull request. Each costly workflow now carries its own observer, so the one-week review covers all of them, and nothing can be skipped until Garry decides after that review.

Blast radius

Five workflow files gain one job each, and no other job, condition or merge tool reads it: test/trunk-dynamic-ci-observers.test.ts asserts that for every workflow (script ran: 58 pass across the four changed test files), and the review traced the check-name matchers in lib/ship-queue-* and scripts/gate-reuse.ts (file:line read). All six observer checks completed green at e3d1feb (live check runs read).

Approvals

No approvals recorded for branch capy/trunk-watches-more-pr-checks-gst-1847.

Summary

Trunk Dynamic CI now watches every costly pull-request workflow, still in learning mode. Each observer runs one pinned action with no checkout and no token scopes, and a contract test keeps all six in that shape. The testing docs say which secrets each observer sees and which skipped runs to leave out when the verdicts are reviewed.

Security read (workflow change)

  • Token scope. TRUNK_API_TOKEN is referenced only in the observer job's env; the test asserts that it appears exactly once per workflow and that no other job mentions Trunk or the token.
  • No fork exposure. The job runs only when github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository. Fork PRs, pushes to main, release, nightly/schedule and workflow_dispatch runs skip it. GitHub also withholds repository secrets from fork pull_request runs, and none of these workflows uses pull_request_target.
  • No PR code beside the token. The job has one step, the SHA-pinned trunk-io/dynamic-ci@5bcddbf (v0.1.7), no checkout, no run:, permissions: {} and github-token: '', and telemetry is off. Trunk reads the diff through its own GitHub App.
  • Can't skip or fail anything. No job needs it, no condition reads it, and it exports no outputs, so switching Trunk to enforcement in its dashboard would still skip nothing. The step and the job both use continue-on-error, the step has a 3-minute timeout inside the job's 5, and a missing token skips the step. Required checks keep their names and results. The one leftover risk is the same one ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) #4012 documents: if GitHub can't fetch the pinned action, that observer's own non-required check shows red.
  • Runner token. permissions: {} removes GITHUB_TOKEN scopes but not the runner's own artifact token, so in evals.yml the third-party action runs beside jobs that pass verdicts through artifacts. The SHA pin freezes its code, which at that commit makes one call to Trunk's API; recorded below as an advisory note.

Test Coverage

No runtime code changed (Coverage: skipped, no runtime file changed). The new observer contract test covers every observer's shape; the three edited workflow tests name the observer explicitly.
Tests ran: free carried from CI (count in the receipt below).

Pre-Landing Review

Pre-Landing Review: 6 issues — 0 auto-fixed, 2 asked (0 fixed, 2 skipped), 0 deferred, 4 noted. None reach the blocking floor (would fail a test, break production, or lose data or secrets).
REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen ledgers_read=4 dispatches=4
Review plan: class gate, depth full, 5 dispatches (batched specialists for testing, security and state machines; red team; native adversarial; outside voice; targeted gate reader), reviewed tree f33af2e40581 at e3d1feb. The later commit cebb98e changes only docs/TESTING_INTERNALS.md (routed docs-only, no new code review).
Gate edits: 14 listed, 13 read, 0 findings — 0 kept, 0 restored, 0 open. The five RH-14 hunks are the new observer jobs' continue-on-error, which the diff justifies (the observer is learning-only and the new contract test pins it fail-open); the test hunks add the observer to job lists and replace "no secrets" with an exact one-secret list.
Outside review: openai/gpt-6.1-sol, completed, VERDICT: CLEAN. Targeted gate reader: no findings.

Review notes

Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.

Exploratory QA

Smoke, report-only, 300 s budget, 2 probes, verdict pass (checkpoint 002 in the run's QA directory).

  • capture 001: bun --no-env-file test --timeout=30000 on the four changed test files: 58 pass, 0 fail, 658 expect() calls.
  • capture 002: check runs at e3d1feb: all six Trunk observer checks completed/success; build-image success; eval-slices skipped.
  • Not exercised live: fork PRs, pushes, dispatches, a missing token (static tests only), and a job-setup failure.

Design Review

No frontend files changed — design review skipped.

Eval Results

No prompt-related files changed — evals skipped.

Review bot comments

No review bot comments.

Scope Drift

Scope Check: CLEAN
Intent: let Trunk observe the Windows checks, quality gate and eval jobs on pull requests, learning only.
Delivered: five observer jobs, one contract test, three test updates, a docs section and a changelog fragment.

Plan Completion

Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches).

TODOS

No TODO items completed in this PR.

Documentation

Status: updated. One factual clarification in docs/TESTING_INTERNALS.md, no blockers.

Scope: committed changes from aa46032..e3d1feb (five workflows gain the learning-only trunk-dynamic-ci observer, plus a new observer contract test, test edits and a changelog fragment) and the unstaged edit to docs/TESTING_INTERNALS.md. Nothing was staged or untracked.

Documentation health:

  • docs/TESTING_INTERNALS.md: Updated. The review-date note now says windows-scope skips only the scoped-test and setup Windows jobs and that windows-smoke always runs. The observer table, the secret-scope wording and the test list match the workflows.
  • changelog.d/capy-trunk-watches-more-pr-checks-gst-1847.md: Current, and not edited because the parent owns it.
  • README.md, AGENTS.md, CONTRIBUTING.md: Current. None of them covers Trunk.

Coverage debt: the new observers have reference and explanation coverage in TESTING_INTERNALS.md. No how-to is needed for internal CI. No diagram drift.

Test plan

  • bun run test (free lane): carried from the free-tests.yml CI run at the pushed head.
  • Focused: bun --no-env-file test --timeout=30000 test/trunk-dynamic-ci-observers.test.ts test/evals-train-workflow.test.ts test/windows-native-workflows.test.ts test/windows-path-scope-workflow.test.ts: 58 pass, 0 fail.

GST-1847. Follows #4012 (GST-1400).

Open in Capy

…l requests, learning only (GST-1847)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

garrytan and others added 3 commits November 7, 2026 23:14
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…lding job in Free Tests only, and list its new test

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…g Trunk's verdicts, workflow by workflow

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
@capy-ai
capy-ai Bot merged commit 892c3b1 into main Nov 8, 2026
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant