Skip to content

What evidence clears an upstream candidate to apply? #449

Closed

Description

@possiblyneal

Part of #448

Question

What must a verdict show before a candidate is applied or rejected? Settle it per class: correctness defect, Recall/Injection/storage change, performance claim, feature request. Cover the reproduction form, the suite and typecheck, which eval, what counts as a benefit, and what evidence a reject needs.

Activity

  1. self-assigned this
    on Nov 18, 2026
  2. possiblyneal commented on Nov 18, 2026

    @possiblyneal
    OwnerAuthor

    Resolution: the evidence bar

    Every verdict on this map meets the bar for its class. A candidate that spans classes meets every bar that applies.

    All classes, apply verdict

    • A test in test/, on branch verify/<slug> cut from penguin, that fails on penguin HEAD (show the failing output) and passes with a prototype fix written for this tree. An upstream diff is read for the diagnosis, never copied.
    • npm run typecheck reports 0 errors and npm test passes on the branch.
    • The fork issue moves to ready-for-agent, with the test path and branch in a comment. The YAML rows get status: "verified-here — <test path>", and stay candidate until /ship lands the fix.

    1. Correctness defect (wrong result, lost or corrupt data, crash, leak across projects)

    • The failing reproduction is the evidence; no Recall gain is required. The fix must not regress: npm run eval:gate and npm run eval:gate:rules pass.
    • Where the defect needs state the Operator may have (a size claim, a duplicate share, a frequency), count it on a copy of the live store (VACUUM INTO under tmp/), never on the live file. A defect that is real but has never occurred in the Operator's store is still applied when it loses data; otherwise it is ranked low and the count goes in the verdict.

    2. Change to Recall, Injection, Summaries or storage content (scoring, filtering, what gets stored or injected)

    • All of class 1, plus npm run eval:suite -- --tier fast against a penguin baseline label: no metric loses beyond its tolerance, and the claimed gain shows. If the change only affects a project-scoped or multi-project path that the corpora cannot exercise, say so and rest on the reproduction plus no loss.
    • A tuning or heuristic change (a threshold, a weight, a dedup rule) needs a measured gain, per root CLAUDE.md Evidence, not just no loss. Use eval:arm / eval:compare on the decision key when the fast tier cannot see it.
    • Add a row to docs/experiments.md (source repo, upstream issue named), with a scorecard in docs/benchmarks/, on the verify branch.

    3. Performance claim

    • A benchmark before and after on a seeded store (eval:burst for Injection latency, or a scripted timing at 20k Observations), with host load recorded. Apply only on a gain on a path the Operator hits, such as hooks, /livez, observe or prompt-context.

    4. Feature request

    • First, the feature screening ticket decides whether it addresses a cost the Operator pays (Goal plus Session history). Only those reach a verification ticket, which then meets the class 2 or 3 bar for its benefit.

    Reject verdict (any class)

    • Evidence that it does not apply here: a test or probe that passes on penguin where upstream's fails, a file:line trace showing the path is absent or already guarded, or a measurement showing no gain or a loss. Record it in the YAML as wont-fix with wont_fix_reason naming the evidence. Close the fork issue, if one exists, with the same evidence. A measured reject also gets a rejected ledger row with Retry if.

    Decided, not escalated: I read "fixing a real error" as enough for a correctness fix (class 1) without a Recall gain, and kept root CLAUDE.md's measured-gain rule for tuning changes (class 2). Undoing this costs only a re-check of the class 1 verdicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions