Repository navigation
🌱 Run Maintained and CI-Tests from typed check definitions - #5425
Merged
Merged
Conversation
justaugustus
requested review from
spencerschrock
and removed request for
a team
November 6, 2026 10:07
justaugustus
temporarily deployed
to
integration-test
November 6, 2026 10:07 — with
GitHub Actions
Inactive
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5425 +/- ##
===========================================
+ Coverage 66.80% 79.68% +12.87%
===========================================
Files 230 263 +33
Lines 16602 14897 -1705
===========================================
+ Hits 11091 11870 +779
+ Misses 4808 3026 -1782
+ Partials 703 1 -702 🚀 New features to boost your workflow:
|
`assignRawData` had 5.6% unit coverage, and the check-table work will move the check-to-collector wiring it holds. Pin the current behavior for Maintained and CI-Tests first: which collectors run, how often they call the RepoClient, what lands in `RawResults`, per-check (not per-method) deduplication, and the exact error text. CI-Tests collects through a map, so `CIInfo` order is not deterministic today; the tests sort before comparing rather than fixing that here. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>
Every check repeats the same collect, store, probe, and evaluate body, and `assignRawData` repeats the collector wiring a second time. Add an unexported `definition` that pairs a typed collector with the `RawResults` field it fills, its probes, and its evaluation, and run two checks from it: Maintained (a `CheckRequest` collector) and CI-Tests (a `RepoClient` collector, through an adapter closure). The exported check functions keep their signatures. Differential tests run the previous bodies and the definitions on the same inputs and require identical results, details, raw data, and RepoClient call sequences, with and without a shared `RawResults`. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>
justaugustus
force-pushed
the
simplification/pilot-1
branch
from
November 6, 2026 10:22
32c8e29 to
a121e1e
Compare
justaugustus
temporarily deployed
to
integration-test
November 6, 2026 10:23 — with
GitHub Actions
Inactive
1 of 2 tasks
justaugustus
added a commit
that referenced
this pull request
Nov 6, 2026
Each check's collector is wired twice: once in the check body and once in probe mode's 20-case switch, with nothing keeping them in step. #5425 moved Maintained and CI-Tests to check definitions. This change plans the other 18 checks, has probe mode read the same definitions, and builds the registry from them, with no observable change. The deltas record two current behaviors as tested requirements: checks and probes collect the same raw data (Packaging excepted), and every check a registered probe needs is a registered check. Assisted-by: LLM Signed-off-by: Stephen Augustus <foo@auggie.dev>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Refactor with tests (infra/tests/other). No change to scores, findings, or output.
What is the current behavior?
Every check repeats the same body: collect raw data, store it in
RawResults, run the probes, evaluate, and wraperrors at two points. Probe mode (
--probes) wires the same checks to the same collectors a second time, in the138-line
assignRawDataswitch inpkg/scorecard. That switch carries anolint:gocognit,gocycloand had almost nounit coverage. Adding a check means wiring it in both places.
What is the new behavior (if this is a feature change)?
Behavior is unchanged. Structurally:
definitionincheckspairs a check's collector with theRawResultsfield it fills, its probes,and its evaluation.
define[T]types the collector and the field together, so pairing a collector with the wrongfield fails to compile.
runDefinitionis the one copy of the collect, probe, and evaluate body.Maintained(aCheckRequestcollector) andCI-Tests(aRepoClientcollector, through an adapter closure) runfrom definitions. The exported
checks.Maintainedandchecks.CITestskeep their signatures and become one-linewrappers.
This is a pilot for two checks, sized so the shape can be judged before anything else moves. It is the first step
toward one check table that both check execution and probe mode read, which would let
assignRawDatago.Two commits, each reviewable alone:
populateRawResultsandassignRawData, for both checks). They pass onthe current code and pin which collectors run,
RepoClientcall counts, stored raw data, per-checkdeduplication, stopping at the first error, and exact error text.
scenario runs the old body and the definition on identical inputs and requires identical results, details, raw
data, and
RepoClientcall sequences, with and without a sharedRawResults.Which issue(s) this PR fixes
NONE
Special notes for your reviewer
Main question: does the definition read better than the body it replaces? Compare
checks/ci_tests.gobefore andafter. If it doesn't, this should stop here, and I'd rather hear that now than after more checks move. The two
checks add net lines, because
checks/definition.gois a one-time cost. Code only shrinks once most checks migrateand probe mode reads the definitions.
Out of scope, deliberately:
init(). Probe mode still usesassignRawDatafor these two checks.checksand would change howcheck-registryis implemented, so it would come with an OpenSpec change first.Two existing behaviors the tests surfaced, left unchanged because changing them would alter output:
raw.CITestscollates through a map, soCIInfo, findings, and details are in random order when a repo has morethan one merged PR. The tests sort before comparing.
evaluation.CITestsignores itsnameargument on success paths and uses its ownCheckCITestsconstant. Thecheck name is defined in
checks,checks/evaluation, andinternal/checknames.Verification beyond CI:
SKIP_GINKGO=1 go test -race -count=1 ./...passed locally.seenupdate, dropping the
CITestResultsassignment, storing raw data when the collector fails, and giving CI-Tests thewrong check name.
Rollback: revert the second commit. The first commit's tests pass on the current code either way.
Does this PR introduce a user-facing change?