Repository navigation
fix(skill-evals): stop the issue-family evals failing correct answers - #1434
Merged
Merged
Conversation
Eleven issue-family eval cases failed on main and on branches alike, some every run and some intermittently. None was a skill defect; each was an ambiguous spec, a fixture that did not show what its expected answer claimed, or a grader comparing structure or keywords instead of substance: - triage step 1: the output spec both said "use default on any hard error" and "report the category the invocation would have resolved to"; a malformed key is now explicitly `explicit-key`. - stale-sweep step 4: the skill's template puts the nudge marker on every comment, close-stale included, as the pr-stale-sweep suite already expects. - fix-workflow step 4: the "minimal fix" was the local guard the skill's cause-not-symptom rule warns against; the fixture now fixes the cause. - fix-workflow step 6: violations are graded on the problems found and on no false positives, not on how entries are grouped or labelled. - reproducer step 1: `line_count` is defined (blank lines count) and the expected values made consistent; recording a prompt-injection note is no longer penalised, while the exact block count still proves the injected instruction was not obeyed. - reproducer steps 3 and 7: brittle keyword regexes over free-text rationales are now judge checks of intent; the multiple-blocks case no longer expects the block that "prefer the simplest complete reproducer" rules out. - reproducer step 5.5: the "clean" reproducer opened and wrote the metadata database, so flagging it was defensible; it is now actually clean. Each fixed case passed at least three consecutive runs, and every affected step directory passes in full. No safety check was weakened. Generated-by: Claude Code (Opus 5.5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running the issue-family eval suites on
mainand on #1400 side by side showed eleven cases failing on both heads — some every run, some intermittently. None is a skill defect: each is an ambiguous output spec, a fixture that doesn't show what its expected answer claims, or a grader comparing structure or keywords instead of substance. This fixes them on the eval side only; noSKILL.mdchanges.case-4-malformed-keydefaulton any hard error" and "report what it would have resolved to"explicit-key;defaultonly when no selector was givencase-2-close-stale-cleancase-1,case-3cls=XComEncoder)case-2,case-3violationsgraded as one prose blob; splitting or relabelling failedcase-2,case-4line_countcounted inconsistently across cases; recording the injection attempt was penalisedline_countdefined (blank lines count); structural checks; exactblock_countstill proves the injection was not obeyedcase-3,case-5case-1-cleancase-3Every fixed case passed at least three consecutive
claude -pruns, and every affected step directory passes in full. No safety check was weakened: injection resistance, the not-safe-to-run cases and the no-false-positive checks all still apply.Was generative AI tooling used to co-author this PR?
🤖 Generated with Claude Code