Repository navigation
fix(reflect): a "nothing found" answer needs recall first (#4567) - #5641
Merged
Merged
Conversation
On a low/mid reflect, fresh mental models release the forced lower layers, and the model could then answer "the bank holds nothing about X" off the page layer alone while recall held the facts. Prompt rules (#5250, a bank directive) were measured on the reporter's banks and did not move it. `done` now carries a required boolean, `answer_found`. When the model sets it to false and recall has not run yet, the loop refuses that `done` once, tells the model why, and forces `recall` on the next turn; the next `done` is accepted. A missing or unreadable field reads as found, so models that skip it behave as before. Tests: loop tests for the refused-then-recalled path and for a negative after recall (accepted), the lenient field parsing, every `done` variant requiring the field, and an hs_llm_core test that stages the early negative and checks the real model recalls and answers from the raw fact.
This was referenced Oct 16, 2026
nicoloboschi
added a commit
that referenced
this pull request
Oct 16, 2026
Conflicts: main's "nothing found needs recall first" (#5641) sits beside the forced follow-up search in the tool choice and runs before the done split; perf_counter everywhere (time.time is banned); main's new reflect tests that answered from the free turn's done now script the closing step.
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4567.
Problem
On a
low/midreflect, fresh mental models release the forced lower layers (the short-circuit inagent.py). The model could then answer "the bank holds nothing about X" off the page layer alone, whilerecallon the same bank returned the facts. This hits the coding-agents session-start reflect, which always runslow.Prompt-level fixes did not move it: #5250 and a bank directive were both measured on the reporter's banks (0/10 and 3/10 short-circuited reflects reached
recall).Fix
donegets a required boolean,answer_found. If the model sets it tofalseandrecallhas not run yet, the loop:doneonce, telling the model to runrecallwith the question's key terms,recallon the next turn,done.The flag is checked in code, not left to the model. A missing or unreadable field counts as found, so models that ignore it behave as before.
highbudget and answers that do find something are unchanged.Known limit: a confident, partial answer marked
answer_found: trueis not caught.Tests
done(answer_found=false)→ refused → forcedrecall→ accepted. And a negative afterrecallis accepted with no extra turn."false"strings, missing field,_rawpayload).donevariant (plain, directives, document) requires the field.hs_llm_core: stages the early negative, then the real model writes the recall query and answers from the raw fact (judged). Passed 3/3 locally on gemini-3.1-flash-lite, plus the existing short-circuit real-LLM tests 3/3.Also checked by hand on gemini-3.1-flash-lite (9 runs): the model sets
answer_foundtrue when it answers and false when nothing exists.