Skip to content

fix(retain): a failed chunk no longer replaces the document (#5589) - #5753

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/retain-atomic-document-tracking
Oct 19, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/retain-atomic-document-tracking

Conversation

@nicoloboschi

@nicoloboschi nicoloboschi commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #5589.

Problem

A retain extracts chunks in parallel and writes them in batches. When a chunk failed (embeddings or LLM outage), the other chunks kept going and the final batch was still written. That first write deletes the document's previous memories and overwrites original_text; only then was the error raised. A chunk that extracts 0 facts never calls embeddings, so during an embeddings outage it "succeeded" — and that was enough to leave the document with 0 memories.

Fix (engine/retain/orchestrator.py)

  • The producer stops at the first failed chunk and cancels the remaining extractions (their output would be thrown away anyway).
  • The consumer discards every batch once a chunk has failed — both mid-stream and at the end-of-stream flush.

Known limit: a document large enough to write several batches (>100 chunks by default) can still have its first batch committed before a later chunk fails. Retrying the same content resumes from there (existing recovery path); making that fully atomic would defeat the streaming memory bound.

Tests

  • hindsight-api-slim/tests/test_retain_failed_chunk_keeps_document.py — fails on main (previous memories deleted), passes here.
  • hindsight-system-tests/tests/test_65_provider_outages.py — new blackbox stories (all 4 fail on main):
    • embeddings outage during an update keeps the previous version, and the update lands once the backend is back
    • LLM 503 during an update, same
    • an outage on a first retain leaves no document; the retry yields one copy of each fact
    • async retain during an outage fails, keeps the old version, and retry_operation applies the update whole
  • The system-test stub can now fail embeddings with a given status (stubs.embeddings.fail_with_status).

A retain whose chunk extraction failed still committed the end-of-stream
batch, and that first write deletes the document's previous memories and
overwrites its text before the error was raised. During an embeddings
outage a 0-fact chunk (which never calls embeddings) was enough to wipe
the document.

- The producer stops at the first failed chunk and cancels the rest.
- The consumer discards every batch once a chunk has failed.
- api-slim regression test, plus system stories for embeddings/LLM
  outages during updates, first retains, and async retain + retry.
- The system-test stub can fail embeddings with a given status.
@nicoloboschi
nicoloboschi merged commit a152525 into main Oct 19, 2026
117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/retain-atomic-document-tracking branch October 19, 2026 10:46
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retain commits document tracking (deletes previous units, overwrites original_text) even when chunk extraction failed, leaving 0-unit documents

1 participant