Skip to content

fix(reflect): replace memory ids the model never saw in answers and pages (#5166) - #5302

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/reflect-unverified-ids-5166
Oct 6, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/reflect-unverified-ids-5166

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #5166.

Problem

Reflect checked the structured memory_ids of done against what it retrieved, but kept the answer text exactly as written. So a full UUID that was made up or garbled was stored in the answer and the mental-model page, and returned 404 when looked up. These IDs often share their first 8 characters with a real ID. Two sources:

  • the model writing a wrong ID itself, or
  • the length rewrite garbling a citation that done got right (the case raised in the issue comments).

Fix

  • ToolResultPresenter records every UUID the model was shown. That covers the raw tool results (so real retrieved IDs, cited by alias or in full, pass) and the opening prompt (question, directives, mission).
  • Once the answer is final (after the length rewrite) every UUID outside that set becomes [unverified id]. This runs on the done path (markdown and structured document) and on forced synthesis.
  • A UUID quoted inside a memory, like a request id, was seen, so it stays. Unrelated UUIDs are not rejected.

Tests

Each scripts the model and runs the real run_reflect_agent. All three fail on main (the fake ID survives) and pass with the fix:

  • done cites a real alias plus a fake ID that shares the real one's prefix. The fake is replaced, and the real ID and a quoted request id stay.
  • The length rewrite garbles a citation in a structured document. Both the rendered text and the document blocks are fixed.
  • Forced synthesis cites a fake ID.

There is also a presenter unit test.

Not covered

An ID copied from an older version of a page the model read still passes, because the model did see it. Catching that would mean checking every cited ID against the bank.

…ages (#5166)

Reflect validated the structured memory_ids of done against what it retrieved,
but took the answer text as written. A full UUID the model made up or garbled
(often sharing the first 8 characters with a real one), or that the length
rewrite garbled from a correct citation, was stored in the mental model page
and 404'd when resolved.

- ToolResultPresenter records every UUID the model was shown: the raw tool
  results (so cited real ids pass) and the opening prompt (question, directives).
- After the length rewrite, on both the done path (text and structured document)
  and forced synthesis, any UUID outside that set becomes "[unverified id]".
  A UUID quoted inside a memory, like a request id, is kept.
- Regression tests: made-up id in done, rewrite garbling a document citation,
  forced synthesis, plus a presenter unit test.
@nicoloboschi
nicoloboschi merged commit b13031b into main Oct 6, 2026
230 of 234 checks passed
@nicoloboschi
nicoloboschi deleted the fix/reflect-unverified-ids-5166 branch October 6, 2026 15:09
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hindsight inline citations can reference nonexistent memory IDs

1 participant