Skip to content

fix(consolidation): pin source_fact_ids to the batch and report actions that cite none (#5273) - #5307

Merged
nicoloboschi merged 2 commits into
mainfrom
fix/5273-consolidation-fact-ids
Oct 6, 2026
Merged

nicoloboschi merged 2 commits into
mainfrom
fix/5273-consolidation-fact-ids

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Fixes #5273.

When a model miscopied a fact id in source_fact_ids (truncated UUID, an observation id), consolidation dropped the whole create/update with no log, then reported its facts as no_durable_knowledge and stamped them consolidated. On the reporter's bank that was 206 of 537 actions with 0 errors.

Changes

  • Ids pinned in the schema. _build_response_model now takes the batch's fact ids and emits items.enum (plus minItems: 1 when HINDSIGHT_API_LLM_SUPPORTS_MAX_ITEMS is on) on source_fact_ids for creates and updates. A grammar-constrained provider can no longer emit an id outside the batch. Validation stays list[str], so a provider that ignores the hint can't fail the batch. Checked that the schema survives both the OpenAI strict transform and the Gemini SDK conversion.
  • Drops are visible. An action whose ids match no batch fact is logged at warning level (kind, ids, text) and counted in the job's new unresolved_actions stat, plus a summary warning line.
  • Bounded cache. has_tagged_union used an unbounded lru_cache keyed on the model class. Now that each batch builds its own class, it's capped at 256.
  • Docs: the SUPPORTS_MAX_ITEMS entry now mentions minItems.

Not done: facts behind a discarded action still count as skipped. The bad ids don't say which facts were meant, so they can't be linked back.

Tests: schema unit tests (enum, minItems gating, still accepts foreign ids) and an end-to-end run where one create cites a mangled id → 1 observation created, unresolved_actions == 1.

…ns that cite none (#5273)

A create/update whose source_fact_ids all missed the batch (truncated UUID,
an observation id) was dropped with no trace, and its facts were reported as
no_durable_knowledge and stamped consolidated.

- The per-batch response schema now pins source_fact_ids to the batch's fact
  ids (enum, plus minItems: 1 when array-length keywords are supported), so a
  grammar-constrained provider cannot emit an unresolvable id. Validation
  stays list[str], so a provider ignoring the hint cannot fail the batch.
- A discarded action is logged at warning level and counted in the job's new
  unresolved_actions stat and the run summary.
- has_tagged_union's cache is bounded, since a model is now built per batch.
@nicoloboschi
nicoloboschi merged commit 3adc7eb into main Oct 6, 2026
115 of 117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/5273-consolidation-fact-ids branch October 6, 2026 15:09
This was referenced Oct 6, 2026
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consolidation silently discards create/update actions whose source_fact_ids match no batch fact, and stamps their facts consolidated

1 participant