Skip to content

ci: add SDK freshness check and test runner jobs#26

Draft
romitg2 wants to merge 2 commits intomainfrom
devin/1773226313-ci-freshness-check
Draft

ci: add SDK freshness check and test runner jobs#26
romitg2 wants to merge 2 commits intomainfrom
devin/1773226313-ci-freshness-check

Conversation

@romitg2
Copy link
Member

@romitg2 romitg2 commented Mar 11, 2026

ci: add SDK freshness check and test runner jobs

Summary

Adds two new CI jobs to the existing workflow in .github/workflows/ci.yml:

  1. SDK Freshness Check — Runs bun run generate in packages/cli and then git diff --exit-code packages/cli/src/generated/ to verify the committed generated SDK matches what openapi.json would produce. Fails with an actionable error message if they're out of sync.

  2. Test — Runs bun run test in packages/cli with TZ=UTC. This will execute vitest tests (currently the snapshot tests from PR feat: add endpoint parameter contract snapshot tests for openapi.json #25, and any future tests added to the CLI package).

Both jobs follow the same runner/bun/cache pattern as existing jobs (blacksmith-2vcpu, bun 1.3.0, frozen lockfile).

Also adds vitest as a devDependency and test/test:update scripts to packages/cli/package.json, which the Test CI job requires.

Review & Testing Checklist for Human

  • Type Check was cancelled on the latest CI run. This appears to be a concurrency issue (new push cancelled the in-progress run), but worth confirming Type Check passes by re-running CI or verifying locally with bun run typecheck.
  • Decide if these jobs should be required checks in branch protection. They are added as CI jobs but are not automatically required for merge — you'll need to update branch protection settings if you want them to block PRs.
  • Verify bun run generate is deterministic in CI. If the code generation tool produces non-deterministic output (e.g., different property ordering across runs), the freshness check will be flaky. Worth running it twice in CI to confirm identical output.
  • Job naming: The "Test" job name is generic but only runs packages/cli tests. If other packages add tests later, consider renaming to "CLI Tests" or restructuring to run tests across packages.
  • Dependency on PR feat: add endpoint parameter contract snapshot tests for openapi.json #25: The test job won't have any tests to run until PR feat: add endpoint parameter contract snapshot tests for openapi.json #25 (endpoint parameter contract snapshots) is also merged. With passWithNoTests: true in vitest config, this job will pass silently until then — confirm this is acceptable.
  • Merge order with PR feat: add endpoint parameter contract snapshot tests for openapi.json #25 and test: add SDK-to-CLI coverage audit snapshot tests #27: All 3 PRs independently add vitest + test scripts to packages/cli/package.json. Whichever merges second/third will have package.json and bun.lock conflicts that need resolving.

Suggested test plan

  1. Merge this PR and PR feat: add endpoint parameter contract snapshot tests for openapi.json #25 together (or test on a branch that includes both)
  2. Verify both new CI jobs run and pass on a clean PR
  3. To test the freshness check: manually edit a file in packages/cli/src/generated/, push, and confirm the SDK Freshness Check job fails with the expected error message
  4. To test the test runner: modify a snapshot value and confirm the Test job fails

Notes

Adds two new CI jobs to the existing workflow:

1. SDK Freshness Check: Regenerates the CLI SDK from openapi.json and
   fails if the committed generated files differ. Ensures the SDK is
   always in sync with the spec.

2. Test: Runs vitest tests in packages/cli with TZ=UTC.

Co-Authored-By: romitgabani1 <romitgabani1.work@gmail.com>
@devin-ai-integration
Copy link
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions github-actions bot added the ci Changes to CI/CD workflows and GitHub Actions label Mar 11, 2026
The CI test job requires vitest and test scripts in packages/cli.
Adds vitest as devDependency and test/test:update scripts.

Co-Authored-By: romitgabani1 <romitgabani1.work@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Changes to CI/CD workflows and GitHub Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant