Skip to content

fix(consolidation): prompt rules and safety nets found by the consolidation evals - #5595

Closed
nicoloboschi wants to merge 1 commit into
consolidation-evals-onlyfrom
consolidation-evals
Closed

nicoloboschi wants to merge 1 commit into
consolidation-evals-onlyfrom
consolidation-evals

Conversation

@nicoloboschi

@nicoloboschi nicoloboschi commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #5626 (the eval suite). Draft until the approach is agreed.

What

Prompt rules and server safety nets for consolidation, each one found by a failing scenario in #5626.

  • Prompt rules:
    • recency follows mentioned_at;
    • a state change is an update, not a delete;
    • a change cascades to every observation it affects;
    • counts are kept as their own facet;
    • lookalike names stay separate;
    • chatter is skipped;
    • an update keeps the full text (current_text);
    • every durable fact lands somewhere.
  • Server safety nets:
    • a delete's evidence is folded into the observation it merged into (merged_into), or re-queued;
    • an update aimed at a fact id becomes a create;
    • observation ids the model miscopied are repaired;
    • a lookalike-name guard;
    • facts the model skipped are asked again;
    • a cascade pass for observations the change left untouched;
    • a list of other observations about the same entities is added to the prompt.

It also puts retired-conventions back in the eval's must-pass set.

Results (Gemini 3.1 Flash Lite, 17 scenarios × 3)

Passed Gemini calls Gemini cost
main 49/51 81 $0.136
this PR 51/51 82 $0.122

Tests

  • Unit: test_consolidation_lookalike.py, plus updated delete tests.
  • System: test_22_consolidation_safety_nets.py.

nicoloboschi added a commit that referenced this pull request Oct 15, 2026
…5607)

* feat(config): one decision-model setup with per-operation overrides

Recall (#5284), reflect (#5382) and consolidation (#5595) each wire the
decision model their own way. Set it up once instead:
HINDSIGHT_API_DECISION_MODEL_* globally, with optional
HINDSIGHT_API_{RECALL,REFLECT,CONSOLIDATION}_DECISION_MODEL_* overrides
that fall back to the global value (PROVIDER=none opts an operation out).
config.decision_model_config(op) resolves them; engine/decision_model.py
is the shared /v1/systemone client. The typesafe reranker keeps its own
settings.

* fix(config): review fixes for the decision-model setup

- Move the decision-model constants out of the middle of the reranker blocks.
- Docs and .env.example: the setup is shared by operations that support a
  decision model, not used by every operation.
- Type the per-setting lookup (str | None) instead of Any.
@nicoloboschi
nicoloboschi force-pushed the consolidation-evals branch 2 times, most recently from 5017fc5 to cbc7720 Compare October 15, 2026 13:09
@nicoloboschi nicoloboschi changed the title feat(consolidation): eval suite, prompt + safety-net fixes, optional Jev triage feat(consolidation): eval suite for consolidation, prompt and safety-net fixes Oct 15, 2026
@nicoloboschi nicoloboschi changed the title feat(consolidation): eval suite for consolidation, prompt and safety-net fixes fix(consolidation): prompt rules and safety nets found by the consolidation evals Oct 15, 2026
@nicoloboschi
nicoloboschi changed the base branch from main to consolidation-evals-only October 15, 2026 14:34
@nicoloboschi
nicoloboschi deleted the branch consolidation-evals-only October 15, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant