You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
What evidence clears an upstream candidate to apply? #449
What must a verdict show before a candidate is applied or rejected? Settle it per class: correctness defect, Recall/Injection/storage change, performance claim, feature request. Cover the reproduction form, the suite and typecheck, which eval, what counts as a benefit, and what evidence a reject needs.
Every verdict on this map meets the bar for its class. A candidate that spans classes meets every bar that applies.
All classes, apply verdict
A test in test/, on branch verify/<slug> cut from penguin, that fails on penguin HEAD (show the failing output) and passes with a prototype fix written for this tree. An upstream diff is read for the diagnosis, never copied.
npm run typecheck reports 0 errors and npm test passes on the branch.
The fork issue moves to ready-for-agent, with the test path and branch in a comment. The YAML rows get status: "verified-here — <test path>", and stay candidate until /ship lands the fix.
1. Correctness defect (wrong result, lost or corrupt data, crash, leak across projects)
The failing reproduction is the evidence; no Recall gain is required. The fix must not regress: npm run eval:gate and npm run eval:gate:rules pass.
Where the defect needs state the Operator may have (a size claim, a duplicate share, a frequency), count it on a copy of the live store (VACUUM INTO under tmp/), never on the live file. A defect that is real but has never occurred in the Operator's store is still applied when it loses data; otherwise it is ranked low and the count goes in the verdict.
2. Change to Recall, Injection, Summaries or storage content (scoring, filtering, what gets stored or injected)
All of class 1, plus npm run eval:suite -- --tier fast against a penguin baseline label: no metric loses beyond its tolerance, and the claimed gain shows. If the change only affects a project-scoped or multi-project path that the corpora cannot exercise, say so and rest on the reproduction plus no loss.
A tuning or heuristic change (a threshold, a weight, a dedup rule) needs a measured gain, per root CLAUDE.md Evidence, not just no loss. Use eval:arm / eval:compare on the decision key when the fast tier cannot see it.
Add a row to docs/experiments.md (source repo, upstream issue named), with a scorecard in docs/benchmarks/, on the verify branch.
3. Performance claim
A benchmark before and after on a seeded store (eval:burst for Injection latency, or a scripted timing at 20k Observations), with host load recorded. Apply only on a gain on a path the Operator hits, such as hooks, /livez, observe or prompt-context.
4. Feature request
First, the feature screening ticket decides whether it addresses a cost the Operator pays (Goal plus Session history). Only those reach a verification ticket, which then meets the class 2 or 3 bar for its benefit.
Reject verdict (any class)
Evidence that it does not apply here: a test or probe that passes on penguin where upstream's fails, a file:line trace showing the path is absent or already guarded, or a measurement showing no gain or a loss. Record it in the YAML as wont-fix with wont_fix_reason naming the evidence. Close the fork issue, if one exists, with the same evidence. A measured reject also gets a rejected ledger row with Retry if.
Decided, not escalated: I read "fixing a real error" as enough for a correctness fix (class 1) without a Recall gain, and kept root CLAUDE.md's measured-gain rule for tuning changes (class 2). Undoing this costs only a re-check of the class 1 verdicts.
Activity
possiblyneal commentedon Nov 18, 2026
Resolution: the evidence bar
Every verdict on this map meets the bar for its class. A candidate that spans classes meets every bar that applies.
All classes, apply verdict
test/, on branchverify/<slug>cut frompenguin, that fails onpenguinHEAD (show the failing output) and passes with a prototype fix written for this tree. An upstream diff is read for the diagnosis, never copied.npm run typecheckreports 0 errors andnpm testpasses on the branch.ready-for-agent, with the test path and branch in a comment. The YAML rows getstatus: "verified-here — <test path>", and staycandidateuntil/shiplands the fix.1. Correctness defect (wrong result, lost or corrupt data, crash, leak across projects)
npm run eval:gateandnpm run eval:gate:rulespass.VACUUM INTOundertmp/), never on the live file. A defect that is real but has never occurred in the Operator's store is still applied when it loses data; otherwise it is ranked low and the count goes in the verdict.2. Change to Recall, Injection, Summaries or storage content (scoring, filtering, what gets stored or injected)
npm run eval:suite -- --tier fastagainst apenguinbaseline label: no metric loses beyond its tolerance, and the claimed gain shows. If the change only affects a project-scoped or multi-project path that the corpora cannot exercise, say so and rest on the reproduction plus no loss.CLAUDE.mdEvidence, not just no loss. Useeval:arm/eval:compareon the decision key when the fast tier cannot see it.docs/experiments.md(sourcerepo, upstream issue named), with a scorecard indocs/benchmarks/, on the verify branch.3. Performance claim
eval:burstfor Injection latency, or a scripted timing at 20k Observations), with host load recorded. Apply only on a gain on a path the Operator hits, such as hooks,/livez, observe or prompt-context.4. Feature request
Reject verdict (any class)
penguinwhere upstream's fails, afile:linetrace showing the path is absent or already guarded, or a measurement showing no gain or a loss. Record it in the YAML aswont-fixwithwont_fix_reasonnaming the evidence. Close the fork issue, if one exists, with the same evidence. A measured reject also gets arejectedledger row with Retry if.Decided, not escalated: I read "fixing a real error" as enough for a correctness fix (class 1) without a Recall gain, and kept root
CLAUDE.md's measured-gain rule for tuning changes (class 2). Undoing this costs only a re-check of the class 1 verdicts.