Skip to content

fix(reflect): a "nothing found" answer needs recall first (#4567) - #5641

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/reflect-absence-needs-recall-4567
Oct 16, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/reflect-absence-needs-recall-4567

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #4567.

Problem

On a low/mid reflect, fresh mental models release the forced lower layers (the short-circuit in agent.py). The model could then answer "the bank holds nothing about X" off the page layer alone, while recall on the same bank returned the facts. This hits the coding-agents session-start reflect, which always runs low.

Prompt-level fixes did not move it: #5250 and a bank directive were both measured on the reporter's banks (0/10 and 3/10 short-circuited reflects reached recall).

Fix

done gets a required boolean, answer_found. If the model sets it to false and recall has not run yet, the loop:

  1. refuses that done once, telling the model to run recall with the question's key terms,
  2. forces recall on the next turn,
  3. accepts the next done.

The flag is checked in code, not left to the model. A missing or unreadable field counts as found, so models that ignore it behave as before. high budget and answers that do find something are unchanged.

Known limit: a confident, partial answer marked answer_found: true is not caught.

Tests

  • Loop: short-circuit → done(answer_found=false) → refused → forced recall → accepted. And a negative after recall is accepted with no extra turn.
  • Lenient parsing ("false" strings, missing field, _raw payload).
  • Every done variant (plain, directives, document) requires the field.
  • hs_llm_core: stages the early negative, then the real model writes the recall query and answers from the raw fact (judged). Passed 3/3 locally on gemini-3.1-flash-lite, plus the existing short-circuit real-LLM tests 3/3.

Also checked by hand on gemini-3.1-flash-lite (9 runs): the model sets answer_found true when it answers and false when nothing exists.

On a low/mid reflect, fresh mental models release the forced lower layers,
and the model could then answer "the bank holds nothing about X" off the
page layer alone while recall held the facts. Prompt rules (#5250, a bank
directive) were measured on the reporter's banks and did not move it.

`done` now carries a required boolean, `answer_found`. When the model sets
it to false and recall has not run yet, the loop refuses that `done` once,
tells the model why, and forces `recall` on the next turn; the next `done`
is accepted. A missing or unreadable field reads as found, so models that
skip it behave as before.

Tests: loop tests for the refused-then-recalled path and for a negative
after recall (accepted), the lenient field parsing, every `done` variant
requiring the field, and an hs_llm_core test that stages the early negative
and checks the real model recalls and answers from the raw fact.
@nicoloboschi
nicoloboschi merged commit e001544 into main Oct 16, 2026
233 of 234 checks passed
@nicoloboschi
nicoloboschi deleted the fix/reflect-absence-needs-recall-4567 branch October 16, 2026 08:33
nicoloboschi added a commit that referenced this pull request Oct 16, 2026
Conflicts: main's "nothing found needs recall first" (#5641) sits beside the forced follow-up
search in the tool choice and runs before the done split; perf_counter everywhere (time.time is
banned); main's new reflect tests that answered from the free turn's done now script the closing
step.
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reflect (0.10.0): low/mid-budget short-circuit answers from mental models alone and reports "the bank holds nothing" for facts that recall finds

1 participant