Skip to content

Map: verify upstream candidates before applying #448

Open
23 of 24 issues completed
Open

Description

@possiblyneal

Destination

Every candidate row in docs/upstream-issue-triage.yaml and docs/upstream-pr-triage.yaml, and each of fork issues #435–#447, has a tested verdict. It is either apply, backed by a reproduction that fails on penguin and passes with a prototype fix, plus the measurement its class needs, and handed to /ship as a ready-for-agent issue. Or it is reject, backed by evidence that it does not reproduce here or does not benefit the Operator, and recorded wont-fix in the YAML. No upstream patch is applied on trust: the fork has diverged too far.

Notes

  • Domain: agentmemory-sqlite, the Operator's fork. Read root CLAUDE.md (Goal, Evidence, Testing) and CONTEXT.md for vocabulary.
  • Execution override: a ticket may write a reproduction test and a prototype fix on a throwaway branch verify/<slug> (pushed, never merged), and run evals. It never lands code on main. Landing is /ship's job after the verdict.
  • The evidence bar every verdict must meet is the first decision on this map. Read it before working any verification ticket.
  • An upstream PR diff is read for the diagnosis, never copied as the fix. Write the fix for this tree.
  • A measurement that ends (kept, rejected or inconclusive) adds a row to docs/experiments.md with its scorecard under docs/benchmarks/, on the verify branch.
  • Each verdict updates its YAML rows (status, disposition) and its fork issue (label ready-for-agent, or close as not reproducing), on the verify branch.
  • Skills to consult: diagnosing-bugs for reproductions; grilling + domain-modeling for grilling tickets.
  • Working material from the triage: tmp/upstream-review/ (verdict notes, repro probe probe/redact.ts, issue bodies).

Decisions so far

  • What evidence clears an upstream candidate to apply?: failing repro on penguin plus a prototype fix on verify/<slug>; correctness fixes need no regression on the gates, tuning changes need a measured gain, rejects need a passing probe or a code trace.
  • Verify: replace import empties the store before validating: apply. It reproduces, and validate-first plus dropping keyless rows holds on verify/436-import-validate-first. Correctness only, so no eval:suite (no eval calls import).
  • Verify: auto-forget supersedes Memories across projects: apply. Cross-scope supersession, early image-ref release and the interval parse all reproduce; fixed on verify/437-auto-forget-scope. Latent: nothing in the Operator's store is hit yet.
  • Verify: capture redaction corrupts data and misses JSON secrets: apply. Structured redaction, JSON-shaped secret keys and unclosed <private> blocks fixed on verify/435-capture-redaction; eval:suite unchanged. Surrogate truncation rejected (3 of 105,699 records, none from a hook cut).
  • Verify: lesson recall cuts candidates before the project filter: apply. Filtering before the cut fixed on verify/438-lesson-recall-starvation; eval:suite flat; on the Operator's Lessons the top pick is unchanged and recall lists gain better-scoring same-project Lessons.
  • Verify: observe dedup hashes only the first 500 chars: apply. Full-payload hash on verify/439-dedup-full-hash; 224 of 16,796 of the Operator's prompts would have been dropped; eval:suite flat.
  • Verify: Reflect Insight identity, source attribution and concept bloat: partial. Project-scoped Insight id and failure reporting apply on verify/440-reflect-insight; cited-only sources and concept trimming not applied, since mem:insights is 2.3% of the store here.
  • Which upstream feature requests earn a test?: one of 52 does (synthetic-Observation age limit, now its own ticket); 50 rejected for empty record kinds, levers already rejected, wont-fix classes or no measurable cost.
  • Verify: four write paths skip the search index: apply, latent. Consolidate, auto-page, heal and mesh now index or de-index on verify/442-index-writes; none has written a Memory on the Operator's store.
  • Verify: age-out synthetic Observations (upstream #1566): not applied. A 60-day rule frees ~28 MB of a 598 MB store, and 1,111 of those Observations are still read as search hits; the 30-day replay corpus cannot reach the rule.
  • Verify: export and snapshot drop data a restore needs: partial. Memory relations now go through export and snapshot, with content-keyed rows, on verify/443-relations; eval:suite flat; latent (0 relations stored). The consolidate crash, the import cap, audit export and image refs are rejected or not applied.
  • Verify: whole-scope reloads cost the Operator latency: partial. Session-cap eviction counts the Session's rows before listing them, on verify/469-scope-reloads. Under the cap, observe p50 drops from 255 to 24 ms and /livez max from 328 to 51 ms (5 interleaved rounds). The count and list endpoints are rejected: none lists a whole scope on a hot path.
  • Should unscoped Memories pass a project filter?: apply. Exclude a non-Global Memory with no project from a project filter, no migration (Operator's call), on verify/441-unscoped-memory; eval:suite flat.
  • Verify: project-scoped smart-search returns a short page: apply. A scoped page short of limit from a full window re-searches at 4x, up to 1,600, on verify/444-scoped-short-page; eval:suite flat.
  • Verify: consolidation writes reworded near-duplicate facts: partial apply. An unkeyed reworded fact reinforces its same-project live twin (Jaccard >= 0.5) on verify/445-reworded-dup-facts; 42% of live facts have such a twin; existing rows stay; eval:suite flat.
  • Verify: agent-sdk child sessions load host plugins: apply, latent. Child query() sets settingSources: [], strictMcpConfig, persistSession: false on verify/446-agent-sdk-isolation; the Operator's .env keeps agent-sdk off.
  • Verify: MCP memory_save reports saved from the fallback: partial. The local fallback reply adds localOnly: true on verify/447-save-fallback-queue; queue-and-replay not applied, since the plugin's AGENTMEMORY_URL already makes a down daemon fail loudly.
  • Verify: MCP shim drops write args and dies silently: partial. Array concepts/files kept on the daemon REST MCP path (shared normalizeList) and a clean exit on stdout EPIPE, on verify/463-mcp-shim; Lesson sourceIds and memory_save ttlDays not applied (nothing reads them); crash-log PRs rejected, since Node already prints the stack to stderr.
  • Verify: facets outlive the records they tag: apply. Every Observation, Memory and Lesson delete path also deletes the facets tagging it (deleteFacetsForTargets), on verify/465-facets; latent, since the live store has 0 facet rows. The heal sweep and remove-by-targetId are not applied.
  • Verify: Recall scoring defects: partial. Applied on verify/467-recall-scoring: search results keep a saved Memory's own type, graph scoring keeps each Observation's best path, and mem::consolidate is locked, writes before it demotes, and counts failed calls. The prefix-bonus lever was measured and is inconclusive (pmb +0.021 with an interval touching 0, gate prompt-submit precision −0.033), so it was not applied. The procedural tier is left for an Operator decision.
  • Verify: config and doctor disagreements: apply most. A placeholder API key no longer picks a provider, doctor honours OPENAI_API_KEY_FOR_LLM=false and the agent SDK and stays quiet without a TTY, the fallback chain describes images, and a model:variant name finds its embedding dimensions, on verify/468-config-doctor; all latent. The MAX_TOKENS default and the status label are rejected (the Operator sets MAX_TOKENS; status already reads the right field).
  • Verify: lock gaps in delete, relate and RMW paths: apply most. Every Memory delete path, relate and evolve now take the Memory write lock, so a delete or relate is no longer undone by a concurrent remember; a half-open circuit breaker lets one probe through, not every caller; on verify/464-locks; latent. Session and graph RMW locks are rejected (already covered by obs: and the graph write lock).
  • Verify: hook capture loses or merges Observations: partial. import-jsonl no longer stamps a timestampless line with the import time, on verify/466-hooks; latent. Hook early exit, repeated-prompt dedup and the durable hook spool are rejected: the request is sent before exit, the dedup is deliberate, and the storeless Sessions trace to hosts that never loaded the plugin or to audited eviction, not to lost Observations.
  • Verify: Session Summaries store stray XML tags: apply. A child item closed with the wrong tag no longer fuses with the next item or keeps the stray tag, on verify/472-xml-children; re-parsing the live store's leaked items clears 59 of 59 Summaries and 784 of 792 Observations. Stored records are not repaired; that would be a separate migration.

Not yet specified

  • Nothing. Every question is decided or handed off (see Handoff).

Handoff

Every fix is verified and uncommitted on its verify/<slug> branch. Each is a ready-for-agent issue for /ship, one PR per issue:
#435, #436, #437, #438, #439, #440, #441, #442, #443, #444, #445, #446, #447, #472, #473 (MCP shim), #474 (locks), #475 (facets), #476 (import-jsonl), #477 (Recall scoring), #478 (config and doctor), #479 (session-cap eviction).

Out of scope

  • Landing fixes on main: that is /ship's job, starting from each ready-for-agent issue this map produces.
  • Reporting findings back upstream.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions