Skip to content

fix(skill-evals): stop the issue-family evals failing correct answers - #1434

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/issue-evals-stability
Sep 27, 2026
Merged

potiuk merged 1 commit into
apache:mainfrom
potiuk:fix/issue-evals-stability

Conversation

@potiuk

@potiuk potiuk commented Sep 27, 2026

Copy link
Copy Markdown
Member

Running the issue-family eval suites on main and on #1400 side by side showed eleven cases failing on both heads — some every run, some intermittently. None is a skill defect: each is an ambiguous output spec, a fixture that doesn't show what its expected answer claims, or a grader comparing structure or keywords instead of substance. This fixes them on the eval side only; no SKILL.md changes.

Suite / case Cause Fix
triage step-1 case-4-malformed-key Output spec said both "use default on any hard error" and "report what it would have resolved to" A malformed key is explicit-key; default only when no selector was given
stale-sweep step-4 case-2-close-stale-clean Expected no nudge marker, but the skill's template puts it on every comment (the pr-stale-sweep suite already expects it on close) Expect the marker
fix-workflow step-4 case-1, case-3 The "minimal fix" was the local guard the cause-not-symptom rule warns against Fixture fixes the cause (cls=XComEncoder)
fix-workflow step-6 case-2, case-3 violations graded as one prose blob; splitting or relabelling failed Judge checks: right problems flagged, no false positives; grouping not significant
reproducer step-1 case-2, case-4 line_count counted inconsistently across cases; recording the injection attempt was penalised line_count defined (blank lines count); structural checks; exact block_count still proves the injection was not obeyed
reproducer step-3 case-3, case-5 Keyword regex over free-text rationale; case-5 expected the block "prefer the simplest complete" rules out Judge checks of intent
reproducer step-5.5 case-1-clean The "clean" reproducer opened and wrote the metadata DB, so flagging it was defensible Reproducer is now actually clean
reproducer step-7 case-3 Keyword regex missed valid phrasings ("AttributeError, not KeyError") Judge check; deterministic cited-error check kept

Every fixed case passed at least three consecutive claude -p runs, and every affected step directory passes in full. No safety check was weakened: injection resistance, the not-safe-to-run cases and the no-false-positive checks all still apply.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

🤖 Generated with Claude Code

Eleven issue-family eval cases failed on main and on branches alike, some
every run and some intermittently. None was a skill defect; each was an
ambiguous spec, a fixture that did not show what its expected answer claimed,
or a grader comparing structure or keywords instead of substance:

- triage step 1: the output spec both said "use default on any hard error"
  and "report the category the invocation would have resolved to"; a
  malformed key is now explicitly `explicit-key`.
- stale-sweep step 4: the skill's template puts the nudge marker on every
  comment, close-stale included, as the pr-stale-sweep suite already expects.
- fix-workflow step 4: the "minimal fix" was the local guard the skill's
  cause-not-symptom rule warns against; the fixture now fixes the cause.
- fix-workflow step 6: violations are graded on the problems found and on no
  false positives, not on how entries are grouped or labelled.
- reproducer step 1: `line_count` is defined (blank lines count) and the
  expected values made consistent; recording a prompt-injection note is no
  longer penalised, while the exact block count still proves the injected
  instruction was not obeyed.
- reproducer steps 3 and 7: brittle keyword regexes over free-text rationales
  are now judge checks of intent; the multiple-blocks case no longer expects
  the block that "prefer the simplest complete reproducer" rules out.
- reproducer step 5.5: the "clean" reproducer opened and wrote the metadata
  database, so flagging it was defensible; it is now actually clean.

Each fixed case passed at least three consecutive runs, and every affected
step directory passes in full. No safety check was weakened.

Generated-by: Claude Code (Opus 5.5)
@potiuk potiuk added family:issue issue-* skills substrate:framework-dev Tool substrate: build / validate / eval the framework itself labels Sep 27, 2026
@potiuk
potiuk merged commit 72e4302 into apache:main Sep 27, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

family:issue issue-* skills substrate:framework-dev Tool substrate: build / validate / eval the framework itself

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant