Repository navigation
fix(#2011): fail release case selection when yq cannot read annotations - #2014
Open
fullsend-ai-coder[bot] wants to merge 1 commit into
Open
fullsend-ai-coder[bot] wants to merge 1 commit into
fullsend-ai-coder[bot] wants to merge 1 commit into
Conversation
In EVAL_TIER=release, the case selection loop ran yq inside a command substitution in an `if` condition, so a yq failure (malformed or unreadable annotations.yaml) compared unequal to "true" exactly like release: false. With no case selected the runner printed "No release:true case" and exited 0, so a release check could pass without knowing which cases to run; next to a valid release case it silently ran only the readable subset. Capture yq's output and exit status separately. If yq cannot read an existing annotations.yaml, print the file path and exit 1 before any eval starts (the EXIT trap removes the staged release-cases dir). When every file reads cleanly and none is marked release: true, the existing notice and exit 0 are unchanged. Add an offline regression to eval/run-functional-test.sh: a malformed annotations.yaml beside a valid release case must exit nonzero, name the file, skip no notice, start no execute step and leave no release-cases dir behind. The existing no-release-case test is kept. Verification: run-functional-test.sh with yq v4.47.1 passes 84/84 (run from a temp copy, because the sandbox denies /dev/zero, which an unrelated existing test reads; main fails the same way in-place). The new test fails against the unfixed script. pre-commit could not fetch hook repos, so its hooks were run directly: whitespace/EOF/line-ending, merge-conflict, private-key, shellcheck 0.10.0.1, lint-release-cases and gitleaks (scan-secrets) all pass. make lint (skillsaw) could not run: uvx is not available in the sandbox. Closes #2011 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 10:26 PM UTC · Completed 10:33 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.00 |
|
Risk Assessment: moderate (2/5) DetailsSmall, well-scoped bot change with offline test coverage, but elevated to moderate risk due to high historical churn and regression frequency in eval scripts and lack of a feature flag toggle. |
|
Looks good to me Previous run<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOltdfQ== --> Looks good to me |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
eval/run-functional.shwithEVAL_TIER=releasetreated ayqread failure on a case'sannotations.yamlthe same asrelease: false.yqran inside a command substitution in anifcondition, so its failure was ignored. With no case selected, the runner printed "No release:true case ... skipping functional tests" and exited 0. The release check passed even though the runner never worked out which cases to run. If a valid release case sat next to the malformed file, it silently ran only the readable subset.Changes
eval/run-functional.sh(release case selection only):yq's output is now captured on its own and its exit status checked. Ifyqcannot read an existingannotations.yaml, the runner printsERROR: could not read <path> — cannot select release casesand exits 1 before any eval starts. The existing EXIT trap removes the stagedrelease-cases-*dir. If every file reads cleanly and none is marked for release, the successful skip (notice + exit 0) is unchanged.eval/run-functional-test.sh: new offline regression. A malformedannotations.yaml(release: [) next to a validrelease: truecase must:release-cases-*dir behindThe existing "no release:true case → notice + exit 0" test is kept.
Scoring, retry behavior and CI workflow inputs are untouched.
Testing
yqv4.47.1:release: false: still prints the notice and exits 0.bash eval/run-functional-test.sh: 84 tests, 0 failures. It ran from a temporary copy because the sandbox denies/dev/zero, which an unrelated existing test (BIG_ERR, around line 1366) reads. Unchangedmainstops at the same point when run in place.pre-commitcould not fetch its hook repos (no network), so I ran the hooks directly. These all pass: end-of-file / trailing-whitespace / mixed-line-ending, check-merge-conflict, detect-private-key, shellcheck 0.10.0.1 (repo args), lint-release-cases, and gitleaks viascan-secrets.make lint(skillsaw) was not run becauseuvxis unavailable in the sandbox. It checks skills and agent docs, not these shell scripts.🤖 Generated with Claude Code
Closes #2011
Post-script verification
agent/2011-release-annotation-read-error)bd6748de5c523826455d92584ec7146ec3716fcc..HEAD)