Repository navigation
Let Trunk watch the Windows checks, quality gate and eval jobs on pull requests, learning only (GST-1847) - #4148
Merged
Conversation
…l requests, learning only (GST-1847) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…lding job in Free Tests only, and list its new test Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…g Trunk's verdicts, workflow by workflow Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trunk's learning-mode observer now also records would-run/would-skip verdicts for the Windows checks, the quality gate and the eval jobs on pull requests, so the measurement covers the expensive PR checks and not just the free suite. Nothing reads the verdicts, so every check still runs with its own name and result; this serves the README goal of keeping CI fast and cheap without losing signal.
What changed
The testing docs now list all six observed workflows and say which skipped jobs to leave out at the review date.
Before/after as text: The testing docs now list all six observed workflows and say which skipped jobs to leave out at the review date
windows-smoke.yml,windows-scoped-tests.yml,windows-setup-e2e.yml,quality-gate.ymlandevals.ymleach get the learning-onlytrunk-dynamic-cijob thatfree-tests.ymlhas had since ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) #4012. Trunk answers only for jobs in the workflow that ran the action (the action sendsGITHUB_WORKFLOW_REF's path and the service enumerates that workflow), so one observer infree-tests.ymlcan't cover the others.windows-smoke,windows-scoped-shard,windows-setup+held-install-journey,quality,build-image+eval-slices. All are plain jobs (no reusable-workflowuses:call jobs, which is what gave gbrain WORKFLOW_NOT_RECOGNIZED).Trunk Dynamic CI (learning only, <workflow>), so no two checks at a head share a name.test/trunk-dynamic-ci-observers.test.tsholds all six observers to one shape. The existing workflow tests that pinned exact job lists or "no secrets" for these files now name the observer explicitly (evals-train-workflow,windows-native-workflows,windows-path-scope-workflow).docs/TESTING_INTERNALS.mdlists every observed workflow and job, limits the "only job that sees a secret" claim to free-tests.yml, and says which runs to leave out at the review date for each workflow.Why
Trunk only answered for the free suite, so the review could not say anything about the Windows, quality and eval checks, which cost the most per pull request. Each costly workflow now carries its own observer, so the one-week review covers all of them, and nothing can be skipped until Garry decides after that review.
Blast radius
Five workflow files gain one job each, and no other job, condition or merge tool reads it:
test/trunk-dynamic-ci-observers.test.tsasserts that for every workflow (script ran: 58 pass across the four changed test files), and the review traced the check-name matchers inlib/ship-queue-*andscripts/gate-reuse.ts(file:line read). All six observer checks completed green at e3d1feb (live check runs read).Approvals
No approvals recorded for branch capy/trunk-watches-more-pr-checks-gst-1847.
Summary
Trunk Dynamic CI now watches every costly pull-request workflow, still in learning mode. Each observer runs one pinned action with no checkout and no token scopes, and a contract test keeps all six in that shape. The testing docs say which secrets each observer sees and which skipped runs to leave out when the verdicts are reviewed.
Security read (workflow change)
TRUNK_API_TOKENis referenced only in the observer job'senv; the test asserts that it appears exactly once per workflow and that no other job mentions Trunk or the token.github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository. Fork PRs, pushes to main, release, nightly/schedule andworkflow_dispatchruns skip it. GitHub also withholds repository secrets from forkpull_requestruns, and none of these workflows usespull_request_target.trunk-io/dynamic-ci@5bcddbf(v0.1.7), no checkout, norun:,permissions: {}andgithub-token: '', and telemetry is off. Trunk reads the diff through its own GitHub App.needsit, no condition reads it, and it exports no outputs, so switching Trunk to enforcement in its dashboard would still skip nothing. The step and the job both usecontinue-on-error, the step has a 3-minute timeout inside the job's 5, and a missing token skips the step. Required checks keep their names and results. The one leftover risk is the same one ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) #4012 documents: if GitHub can't fetch the pinned action, that observer's own non-required check shows red.permissions: {}removes GITHUB_TOKEN scopes but not the runner's own artifact token, so inevals.ymlthe third-party action runs beside jobs that pass verdicts through artifacts. The SHA pin freezes its code, which at that commit makes one call to Trunk's API; recorded below as an advisory note.Test Coverage
No runtime code changed (Coverage: skipped, no runtime file changed). The new observer contract test covers every observer's shape; the three edited workflow tests name the observer explicitly.
Tests ran: free carried from CI (count in the receipt below).
Pre-Landing Review
Pre-Landing Review: 6 issues — 0 auto-fixed, 2 asked (0 fixed, 2 skipped), 0 deferred, 4 noted. None reach the blocking floor (would fail a test, break production, or lose data or secrets).
REVIEW_LOOP: passes=1 blocking_found=0 deferred=0 noted=0 fixed=0 scope=frozen ledgers_read=4 dispatches=4
Review plan: class gate, depth full, 5 dispatches (batched specialists for testing, security and state machines; red team; native adversarial; outside voice; targeted gate reader), reviewed tree f33af2e40581 at e3d1feb. The later commit cebb98e changes only docs/TESTING_INTERNALS.md (routed docs-only, no new code review).
Gate edits: 14 listed, 13 read, 0 findings — 0 kept, 0 restored, 0 open. The five RH-14 hunks are the new observer jobs'
continue-on-error, which the diff justifies (the observer is learning-only and the new contract test pins it fail-open); the test hunks add the observer to job lists and replace "no secrets" with an exact one-secret list.Outside review: openai/gpt-6.1-sol, completed, VERDICT: CLEAN. Targeted gate reader: no findings.
Review notes
Below the blocking floor (advisory, informational, test-gap or wording items); recorded here and not fixed in this PR.
.github/workflows/evals.yml:85A "Set up job" failure of an observer (action download, deleted pinned commit) shows a red non-required check thatgstack-ship-queue readycounts; now six checks per PR instead of one. Documented, same as free-tests.yml since ci: Trunk Dynamic CI observes free-suite and macOS regressions, learning only (GST-1400) #4012; fixing it is pr-wait tooling, which is frozen. (red-team at dc25d41, native adversarial at e3d1feb).github/workflows/evals.yml:80Dependabot PRs now get one evals observer check (previously no evals check); its step skips without the secret but shares the setup-failure exposure. Fix if wanted: addgithub.actor != 'dependabot[bot]'to the observer'sif:. (red-team refactor: reorganize codebase into modular structure #2, native adversarial Project-local .claude/skills/gstack install breaks /browse and setup can report false success #3).github/workflows/evals.yml:98eval-slicesnever runs on pull-request pushes under the train policy (pr-summary mode; capture 002 shows it skipped), so its observer verdict has no job result to compare against. The docs now say to leave those runs out at the review date; droppingeval-slicesfromjob-keysis the alternative. (red-team docs: add README and CLAUDE.md #1).github/workflows/evals.yml:69permissions: {}does not remove the runner's ACTIONS_RUNTIME_TOKEN, so a compromised action could pre-register an artifact name later evals.yml jobs download. Theoretical while the action is SHA-pinned; revisit if it starts fetching code at runtime. (native adversarial refactor: reorganize codebase into modular structure #2)docs/TESTING_INTERNALS.md"nothing in this workflow reads" was singular for six workflows; the docs audit changed it to "its workflow". (native adversarial Fix project-local /browse installs and harden setup #4, fixed by the docs audit)Exploratory QA
Smoke, report-only, 300 s budget, 2 probes, verdict pass (checkpoint 002 in the run's QA directory).
bun --no-env-file test --timeout=30000on the four changed test files: 58 pass, 0 fail, 658 expect() calls.build-imagesuccess;eval-slicesskipped.Design Review
No frontend files changed — design review skipped.
Eval Results
No prompt-related files changed — evals skipped.
Review bot comments
No review bot comments.
Scope Drift
Scope Check: CLEAN
Intent: let Trunk observe the Windows checks, quality gate and eval jobs on pull requests, learning only.
Delivered: five observer jobs, one contract test, three test updates, a docs section and a changelog fragment.
Plan Completion
Plan completion audit: not run (no plan is bound to this branch and no docs/designs/ file matches).
TODOS
No TODO items completed in this PR.
Documentation
Status: updated. One factual clarification in docs/TESTING_INTERNALS.md, no blockers.
Scope: committed changes from aa46032..e3d1feb (five workflows gain the learning-only
trunk-dynamic-ciobserver, plus a new observer contract test, test edits and a changelog fragment) and the unstaged edit to docs/TESTING_INTERNALS.md. Nothing was staged or untracked.Documentation health:
windows-scopeskips only the scoped-test and setup Windows jobs and thatwindows-smokealways runs. The observer table, the secret-scope wording and the test list match the workflows.Coverage debt: the new observers have reference and explanation coverage in TESTING_INTERNALS.md. No how-to is needed for internal CI. No diagram drift.
Test plan
bun run test(free lane): carried from the free-tests.yml CI run at the pushed head.bun --no-env-file test --timeout=30000 test/trunk-dynamic-ci-observers.test.ts test/evals-train-workflow.test.ts test/windows-native-workflows.test.ts test/windows-path-scope-workflow.test.ts: 58 pass, 0 fail.GST-1847. Follows #4012 (GST-1400).