Skip to content

🌱 Run Maintained and CI-Tests from typed check definitions - #5425

Merged
justaugustus merged 2 commits into
ossf:mainfrom
justaugustus:simplification/pilot-1
Nov 6, 2026
Merged

justaugustus merged 2 commits into
ossf:mainfrom
justaugustus:simplification/pilot-1

Conversation

@justaugustus

Copy link
Copy Markdown
Member

What kind of change does this PR introduce?

Refactor with tests (infra/tests/other). No change to scores, findings, or output.

What is the current behavior?

Every check repeats the same body: collect raw data, store it in RawResults, run the probes, evaluate, and wrap
errors at two points. Probe mode (--probes) wires the same checks to the same collectors a second time, in the
138-line assignRawData switch in pkg/scorecard. That switch carries a nolint:gocognit,gocyclo and had almost no
unit coverage. Adding a check means wiring it in both places.

What is the new behavior (if this is a feature change)?

Behavior is unchanged. Structurally:

  • An unexported definition in checks pairs a check's collector with the RawResults field it fills, its probes,
    and its evaluation. define[T] types the collector and the field together, so pairing a collector with the wrong
    field fails to compile.
  • runDefinition is the one copy of the collect, probe, and evaluate body.
  • Maintained (a CheckRequest collector) and CI-Tests (a RepoClient collector, through an adapter closure) run
    from definitions. The exported checks.Maintained and checks.CITests keep their signatures and become one-line
    wrappers.

This is a pilot for two checks, sized so the shape can be judged before anything else moves. It is the first step
toward one check table that both check execution and probe mode read, which would let assignRawData go.

  • Tests for the changes have been added (for bug fixes/features)

Two commits, each reviewable alone:

  1. Characterization tests for probe mode (populateRawResults and assignRawData, for both checks). They pass on
    the current code and pin which collectors run, RepoClient call counts, stored raw data, per-check
    deduplication, stopping at the first error, and exact error text.
  2. The migration, plus differential tests. The previous check bodies are kept verbatim in the test file. Each
    scenario runs the old body and the definition on identical inputs and requires identical results, details, raw
    data, and RepoClient call sequences, with and without a shared RawResults.

Which issue(s) this PR fixes

NONE

Special notes for your reviewer

Main question: does the definition read better than the body it replaces? Compare checks/ci_tests.go before and
after. If it doesn't, this should stop here, and I'd rather hear that now than after more checks move. The two
checks add net lines, because checks/definition.go is a one-time cost. Code only shrinks once most checks migrate
and probe mode reads the definitions.

Out of scope, deliberately:

  • Registration stays in each check's init(). Probe mode still uses assignRawData for these two checks.
  • Making probe mode read the definitions would need an exported accessor in checks and would change how
    check-registry is implemented, so it would come with an OpenSpec change first.

Two existing behaviors the tests surfaced, left unchanged because changing them would alter output:

  • raw.CITests collates through a map, so CIInfo, findings, and details are in random order when a repo has more
    than one merged PR. The tests sort before comparing.
  • evaluation.CITests ignores its name argument on success paths and uses its own CheckCITests constant. The
    check name is defined in checks, checks/evaluation, and internal/checknames.

Verification beyond CI:

  • SKIP_GINKGO=1 go test -race -count=1 ./... passed locally.
  • Mutations were applied one at a time and reverted. Each one fails the new tests: dropping the probe-mode seen
    update, dropping the CITestResults assignment, storing raw data when the collector fails, and giving CI-Tests the
    wrong check name.

Rollback: revert the second commit. The first commit's tests pass on the current code either way.

Does this PR introduce a user-facing change?

NONE

@justaugustus
justaugustus requested a review from a team as a code owner November 6, 2026 10:07
@justaugustus
justaugustus requested review from spencerschrock and removed request for a team November 6, 2026 10:07
@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.68%. Comparing base (353ed60) to head (a121e1e).
⚠️ Report is 466 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5425       +/-   ##
===========================================
+ Coverage   66.80%   79.68%   +12.87%     
===========================================
  Files         230      263       +33     
  Lines       16602    14897     -1705     
===========================================
+ Hits        11091    11870      +779     
+ Misses       4808     3026     -1782     
+ Partials      703        1      -702     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

`assignRawData` had 5.6% unit coverage, and the check-table work will
move the check-to-collector wiring it holds. Pin the current behavior
for Maintained and CI-Tests first: which collectors run, how often
they call the RepoClient, what lands in `RawResults`, per-check (not
per-method) deduplication, and the exact error text.

CI-Tests collects through a map, so `CIInfo` order is not
deterministic today; the tests sort before comparing rather than
fixing that here.

Assisted-by: LLM
Signed-off-by: Stephen Augustus <foo@auggie.dev>
Every check repeats the same collect, store, probe, and evaluate body,
and `assignRawData` repeats the collector wiring a second time. Add an
unexported `definition` that pairs a typed collector with the
`RawResults` field it fills, its probes, and its evaluation, and run
two checks from it: Maintained (a `CheckRequest` collector) and
CI-Tests (a `RepoClient` collector, through an adapter closure).

The exported check functions keep their signatures. Differential
tests run the previous bodies and the definitions on the same inputs
and require identical results, details, raw data, and RepoClient call
sequences, with and without a shared `RawResults`.

Assisted-by: LLM
Signed-off-by: Stephen Augustus <foo@auggie.dev>
@justaugustus justaugustus moved this to In Progress in OpenSSF Scorecard Nov 6, 2026
@justaugustus justaugustus added this to the scorecard-simplification milestone Nov 6, 2026
@justaugustus
justaugustus force-pushed the simplification/pilot-1 branch from 32c8e29 to a121e1e Compare November 6, 2026 10:22
@justaugustus
justaugustus merged commit 3c43c65 into ossf:main Nov 6, 2026
20 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in OpenSSF Scorecard Nov 6, 2026
justaugustus added a commit that referenced this pull request Nov 6, 2026
Each check's collector is wired twice: once in the check body and once
in probe mode's 20-case switch, with nothing keeping them in step.
#5425 moved Maintained and CI-Tests to check definitions. This change
plans the other 18 checks, has probe mode read the same definitions,
and builds the registry from them, with no observable change.

The deltas record two current behaviors as tested requirements: checks
and probes collect the same raw data (Packaging excepted), and every
check a registered probe needs is a registered check.

Assisted-by: LLM
Signed-off-by: Stephen Augustus <foo@auggie.dev>

This branch was previously deployed

2 inactive deployments
gitlab — a121e1ea Deployed Nov 6, 2026 by justaugustus via gitlab-integration-trusted #5646
integration-test — a121e1ea Deployed Nov 6, 2026 by justaugustus via integration-trusted #12953
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant