Repository navigation
feat(config): one decision-model setup with per-operation overrides - #5607
Merged
Merged
Conversation
Recall (#5284), reflect (#5382) and consolidation (#5595) each wire the decision model their own way. Set it up once instead: HINDSIGHT_API_DECISION_MODEL_* globally, with optional HINDSIGHT_API_{RECALL,REFLECT,CONSOLIDATION}_DECISION_MODEL_* overrides that fall back to the global value (PROVIDER=none opts an operation out). config.decision_model_config(op) resolves them; engine/decision_model.py is the shared /v1/systemone client. The typesafe reranker keeps its own settings.
- Move the decision-model constants out of the middle of the reranker blocks. - Docs and .env.example: the setup is shared by operations that support a decision model, not used by every operation. - Type the per-setting lookup (str | None) instead of Any.
nicoloboschi
added a commit
that referenced
this pull request
Oct 15, 2026
…o more "fast" Reflect is decision-orchestrated whenever a decision model is set up for it (HINDSIGHT_API_DECISION_MODEL_*, #5607), instead of a reflect_mode=fast bank setting that read the TypeSafe reranker key. Two per-bank switches, on by default, opt out: reflect_use_decision_model and mental_model_refresh_use_decision_model. - run_reflect_agent loses its fast flag: an evidence_is_sufficient_fn is the mode. - Trace scopes: round_query, decision_prune, decision_sufficiency, decision_pages_sufficiency. - The setup-to-post timing joins the closing [REFLECT] Complete line. - Shortlist and keep floor are constructor arguments, not instance mutation. - Client wrappers and bank templates accept both switches.
This was referenced Oct 15, 2026
nicoloboschi
added a commit
that referenced
this pull request
Oct 16, 2026
… set up (#5382) * feat(reflect): fast mode — parallel retrieval rounds, decision-model pruning and sufficiency reflect_mode=fast (per bank): one LLM call rewrites the request into a search query (or finds nothing to search and returns empty), every retrieval layer runs at once with it, a decision model (TypeSafe System One) prunes the evidence and judges whether it answers the request. Enough: the answer is the next LLM call. Not enough: the LLM writes the next query and another round runs. A refused or failing decision model hands stopping back to the LLM. The reflect prompt tells decisions from states: a later change without its own rationale is a departure from a recorded decision, not its replacement. * test(reflect): the expected tools prompt carries the decision-vs-state rule * fix(reflect): every answer is written by the closing step, never taken from a free turn's done A free (auto) turn that called done had its answer taken as-is, without the closing request that restates the question and asks for every fact and exact value. On sde-bench seqledger-history that dropped a rule's return value (present three times in the evidence), and the coding agent needed a correction. The free turn's done now hands over to the closing step, as the end of the loop and fast mode's 'enough' verdict already do. * refactor(reflect): the decision-model setup picks the reflect mode; no more "fast" Reflect is decision-orchestrated whenever a decision model is set up for it (HINDSIGHT_API_DECISION_MODEL_*, #5607), instead of a reflect_mode=fast bank setting that read the TypeSafe reranker key. Two per-bank switches, on by default, opt out: reflect_use_decision_model and mental_model_refresh_use_decision_model. - run_reflect_agent loses its fast flag: an evidence_is_sufficient_fn is the mode. - Trace scopes: round_query, decision_prune, decision_sufficiency, decision_pages_sufficiency. - The setup-to-post timing joins the closing [REFLECT] Complete line. - Shortlist and keep floor are constructor arguments, not instance mutation. - Client wrappers and bank templates accept both switches. * test(reflect): each operation's decision-model switch, in the engine and end to end Unit: reflect_async hands the agent the decision model only when one is set up and the bank keeps that operation's switch on (reflect vs. knowledge-page refresh, dry run too). System: a refresh follows mental_model_refresh_use_decision_model, not reflect's switch. * fix(system-evals): wait_until_settled(allow_failed=True), which test_09 calls test_09 refreshes each imported bank's pages with a real model; a refresh can fail once and be retried by the worker, and the staleness check after it is the gate. The option was lost when the branch was squashed. * feat(reflect): reflect follows the bank mission, and decision mode searches after "not enough" - The bank mission is restated after the question in the opening user message (and shown there in the prompt preview): as the system prompt's role line it lost to the retrieval ladder, and agent turns answered from a page that had misread the decision. - After the first "not enough" verdict the next turn is forced to search, instead of refusing its done and paying for the refused turn. - With the decision model, the free turn's done answers (documents still go through the closing step); a done that does not parse is handed back with the error. - Delta page refresh prompt: a dated example of backfilled history. - test_09: every bank gets a reflect mission ranking its sources; judge criteria accept an answer that reports the decision plus a later departure, and "no decision" needs both halves. Review fixes: a test for the malformed-done hand-back in decision mode; the rejection counter's comment now says it covers the payload too. * fix(reflect): review findings — decision model provider, query rewrite, fresh pages, follow-up calls - Reflect's decision mode speaks only TypeSafe's System One: another provider (cloudflare, openai) now leaves reflect on agent turns with one warning, instead of failing every first round. Documented. - The search-query rewrite: runs at the mechanical-pass temperature (unset for reasoning models), reads the caller's context, falls back to the request as written when the call fails, and a document (refresh) never searches for the literal "NONE". - In a decision-orchestrated round, fresh pages the decision model found not to answer no longer release forcing nor tell the LLM to answer from them alone. - A follow-up turn's other calls (expand, get_document, read_mental_models) run with its round instead of being dropped. - Round recall never goes below the bank's own recall size; a pruned observations result reports its pruned count; the decision switch is read directly (no getattr default). - "Latest wins" now names its one exception, a decision, and the decision rule carries an example: without it the model reasoned the cleanup overturned nothing and still called it the rule. - New hs_llm_core judge tests: the mission sends reflect past a fresh page that misread a decision, and a later cleanup does not replace a decision (both 4/4 on gemini-3.1-flash-lite). - System tests: the stub decision model is set up for reflect only (globally it switched recall to decision reranking and broke the reranker stories); the document re-ask and expand-citation stories script the closing step. - Regenerated clients, bank-template schema and docs skill. * test(reflect): round recall keeps the bank's recall size; drop a no-op count fix Second review pass: a test pins the round recall floor (fails without it). The pruned observation count was never shown to the model (the presenter drops count), so that fix is removed rather than tested. A comment named a tool reflect does not have. * fix(reflect): a bank recall size above the argument cap stays uncapped in decision rounds Third review pass: the round passed max(8192, bank size) as an argument, and an argument is capped at 16k like a model-requested size, so a bank set to 20000 got 16000 in decision rounds. The round now raises the size only when the bank's is smaller; the test uses 20000. * test(reflect): the round widens recall only for the decision model's cut Fourth review pass: a test pins the widening (and that no cut means the bank's size); the comment says a passed size still yields to a nearly full context, and the larger-size test is renamed to what it checks. * test(reflect): the token-limit tests' stub config carries the decision-model switches Reading the switch directly (no getattr default) surfaced that this stub config never had them.
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Recall (#5284), reflect (#5382) and consolidation (#5595) each set up the decision model their own way. This adds one setup they will all share. It changes no behaviour yet: the three feature PRs rebase onto it and each adds its own on/off switch.
HINDSIGHT_API_DECISION_MODEL_{PROVIDER,API_KEY,MODEL,BASE_URL,TIMEOUT,MAX_CONCURRENT}:none(default) ortypesafe.HINDSIGHT_API_{RECALL,REFLECT,CONSOLIDATION}_DECISION_MODEL_{PROVIDER,API_KEY,MODEL,BASE_URL}: per-operation overrides, each falling back to the global value, likeREFLECT_LLM_*→LLM_*.PROVIDER=noneopts one operation out; an operation can also have one without a global setup.config.decision_model_config(op)returns the resolved setup, orNone. Unknown provider or missing key fails at startup.engine/decision_model.py: shared/v1/systemoneclient, reply parsed into a typed model.The
typesafereranker (HINDSIGHT_API_RERANKER_TYPESAFE_*) is left as it is. Using Jev as a cross-encoder stays a separate choice.Next
Each feature PR adds
{op}_use_decision_model(per bank, on by default = used when a decision model is set up). For reflect, on means fast mode.Tests
tests/test_decision_model_config.py: default none, global reaches every operation, per-setting fallback, opt-out, operation-only setup, startup errors, credential fields, client request/reply.