Repository navigation
Conversation
|
Follow-up experiment (not part of this PR): a throwaway server overlay ran exact-identifier recall and model-formulated contextual recall deterministically, interleaved their results, and deduplicated memory/chunk IDs before synthesis. Same isolated bank snapshot, original hook prompt, Codex provider/model, and no added retrieval instructions. All 3 runs executed both recall branches with no duplicate memory IDs in their merged result. Two answers recovered both detailed issue scope and known delivery history; one still missed the delivery record because neither branch retrieved it. Times: 70.07 / 46.41 / 38.68 seconds (median 46.41s), compared with 117.40s median for the prompt-only combined experiment. Three visible retrieval calls per run, rather than repeated page/observation searches. Serialized tool output remained large: approximately 231–261k characters. Caveats: only 3 trials; manually evaluated; model-requested per-query budgets varied; the union can exceed a single query budget. This does not yet meet the reliable-coverage criterion and is not ready to become production code. The next investigation is identifier-aware candidate selection/ranking plus a shared retrieval budget. This PR remains a narrowly scoped partial fix for #4567. |
…-tests The startup sweep of leftover `systest-` banks lived in the session fixture, so the refresh-dedupe story's own database was never swept. That database is the one most likely to be left dirty: a run killed while a refresh is deliberately held leaves that operation behind for the next server's worker to resume, with an LLM call no test declared. Hoist the sweep into `start_hindsight_server` so it runs for whichever database a server points at. Also move the response-gate self-test under `tests/harness/` (it tests the stub, not Hindsight), document the second pg0 instance and the response gate in the README, and say why the dedupe fixture zeroes the consolidation reservation.
Summary
82081693a). Production deduplication behavior is unchanged.Type of change
Trade-off and scope
Restores lower-layer retrieval even for page-answerable questions, adding retrieval/LLM-turn latency. Context-size and timeout work in #4566 and #4568 is out of scope. Retrieval-query experiments described below are not included in this PR.
Test plan
These are local validation results, not a claim that the latest GitHub Actions run has passed.
Partially addresses #4567
CI test stabilization
The blackbox CI failure was in the existing refresh-deduplication test: it expected at most two cumulative operations after five sequential requests. That assumes the first requested refresh remains pending throughout the burst. Explicit refreshes deliberately deduplicate pending work, not processing work: a running refresh may have read its inputs before the caller's edit, so a queued successor is valid.
The revised tests use an isolated database and a real one-slot worker, holding its processing refresh at the stub LLM with a cross-thread response gate. They assert that all five new requests return one distinct pending operation ID, with exactly one processing refresh and one pending successor. Cleanup releases the gate and awaits background completion. No sleeps, disabled workers, or production deduplication changes are used. The gate has tests for cross-event-loop coordination, release after failure, and rejection of nested gates.
Deployment findings and remaining gap
Candidate commit
c6efe5458was built into a separate Docker image. The selected bank was copied with embeddings, links, and mental-model freshness preserved; the candidate worker was disabled and its LLM timeout set to 180 seconds. The live server was not modified. This is a current-bank reproduction, not an A/B comparison against the PR base on the historical data.The forced-retrieval restoration works, but query formulation and evidence coverage remain unresolved. Neither prompt-only coordination nor the deterministic dual-query prototype reliably recovered all known evidence. Identifier-aware candidate selection/ranking and a shared retrieval budget need further investigation in a separate follow-up; no prototype code is included here. No private bank data or answer excerpts are attached.