Skip to content

fix(#2011): fail release case selection when yq cannot read annotations - #2014

Open
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/2011-release-annotation-read-error
Open

fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/2011-release-annotation-read-error

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

eval/run-functional.sh with EVAL_TIER=release treated a yq read failure on a case's annotations.yaml the same as release: false. yq ran inside a command substitution in an if condition, so its failure was ignored. With no case selected, the runner printed "No release:true case ... skipping functional tests" and exited 0. The release check passed even though the runner never worked out which cases to run. If a valid release case sat next to the malformed file, it silently ran only the readable subset.

Changes

  • eval/run-functional.sh (release case selection only): yq's output is now captured on its own and its exit status checked. If yq cannot read an existing annotations.yaml, the runner prints ERROR: could not read <path> — cannot select release cases and exits 1 before any eval starts. The existing EXIT trap removes the staged release-cases-* dir. If every file reads cleanly and none is marked for release, the successful skip (notice + exit 0) is unchanged.

  • eval/run-functional-test.sh: new offline regression. A malformed annotations.yaml (release: [) next to a valid release: true case must:

    • exit nonzero
    • name the file
    • not print the skip notice
    • not start the execute step
    • not leave a release-cases-* dir behind

    The existing "no release:true case → notice + exit 0" test is kept.

Scoring, retry behavior and CI workflow inputs are untouched.

Testing

  • Ran the issue's reproduction with yq v4.47.1:
    • Malformed file: now prints the parser error and the file path, then exits 1.
    • Valid release: false: still prints the notice and exits 0.
  • bash eval/run-functional-test.sh: 84 tests, 0 failures. It ran from a temporary copy because the sandbox denies /dev/zero, which an unrelated existing test (BIG_ERR, around line 1366) reads. Unchanged main stops at the same point when run in place.
  • The new test fails against the unfixed script.
  • pre-commit could not fetch its hook repos (no network), so I ran the hooks directly. These all pass: end-of-file / trailing-whitespace / mixed-line-ending, check-merge-conflict, detect-private-key, shellcheck 0.10.0.1 (repo args), lint-release-cases, and gitleaks via scan-secrets.
  • make lint (skillsaw) was not run because uvx is unavailable in the sandbox. It checks skills and agent docs, not these shell scripts.

🤖 Generated with Claude Code


Closes #2011

Post-script verification

  • Branch is not main/master (agent/2011-release-annotation-read-error)
  • Secret scan passed (gitleaks — bd6748de5c523826455d92584ec7146ec3716fcc..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Final PR description secret scan passed (gitleaks — no-git)

In EVAL_TIER=release, the case selection loop ran yq inside a command
substitution in an `if` condition, so a yq failure (malformed or
unreadable annotations.yaml) compared unequal to "true" exactly like
release: false. With no case selected the runner printed "No
release:true case" and exited 0, so a release check could pass without
knowing which cases to run; next to a valid release case it silently
ran only the readable subset.

Capture yq's output and exit status separately. If yq cannot read an
existing annotations.yaml, print the file path and exit 1 before any
eval starts (the EXIT trap removes the staged release-cases dir). When
every file reads cleanly and none is marked release: true, the
existing notice and exit 0 are unchanged.

Add an offline regression to eval/run-functional-test.sh: a malformed
annotations.yaml beside a valid release case must exit nonzero, name
the file, skip no notice, start no execute step and leave no
release-cases dir behind. The existing no-release-case test is kept.

Verification: run-functional-test.sh with yq v4.47.1 passes 84/84
(run from a temp copy, because the sandbox denies /dev/zero, which an
unrelated existing test reads; main fails the same way in-place). The
new test fails against the unfixed script. pre-commit could not fetch
hook repos, so its hooks were run directly: whitespace/EOF/line-ending,
merge-conflict, private-key, shellcheck 0.10.0.1, lint-release-cases
and gitleaks (scan-secrets) all pass. make lint (skillsaw) could not
run: uvx is not available in the sandbox.

Closes #2011

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner November 13, 2026 22:24
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Nov 13, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:26 PM UTC · Completed 10:33 PM UTC

Commit: 6332e47 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.00

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Nov 13, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Small, well-scoped bot change with offline test coverage, but elevated to moderate risk due to high historical churn and regression frequency in eval scripts and lack of a feature flag toggle.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Looks good to me

Previous run

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOltdfQ== -->

Looks good to me

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Nov 13, 2026

This branch was previously deployed

1 inactive deployment
dev — 6332e47d Deployed Nov 13, 2026 by fullsend-ai-coder[bot] via behaviour #2055
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eval: fail release case selection when yq cannot read annotations

1 participant