Repository navigation
fix(retain): a failed chunk no longer replaces the document (#5589) - #5753
Merged
Merged
Conversation
A retain whose chunk extraction failed still committed the end-of-stream batch, and that first write deletes the document's previous memories and overwrites its text before the error was raised. During an embeddings outage a 0-fact chunk (which never calls embeddings) was enough to wipe the document. - The producer stops at the first failed chunk and cancels the rest. - The consumer discards every batch once a chunk has failed. - api-slim regression test, plus system stories for embeddings/LLM outages during updates, first retains, and async retain + retry. - The system-test stub can fail embeddings with a given status.
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5589.
Problem
A retain extracts chunks in parallel and writes them in batches. When a chunk failed (embeddings or LLM outage), the other chunks kept going and the final batch was still written. That first write deletes the document's previous memories and overwrites
original_text; only then was the error raised. A chunk that extracts 0 facts never calls embeddings, so during an embeddings outage it "succeeded" — and that was enough to leave the document with 0 memories.Fix (
engine/retain/orchestrator.py)Known limit: a document large enough to write several batches (>100 chunks by default) can still have its first batch committed before a later chunk fails. Retrying the same content resumes from there (existing recovery path); making that fully atomic would defeat the streaming memory bound.
Tests
hindsight-api-slim/tests/test_retain_failed_chunk_keeps_document.py— fails on main (previous memories deleted), passes here.hindsight-system-tests/tests/test_65_provider_outages.py— new blackbox stories (all 4 fail on main):retry_operationapplies the update wholestubs.embeddings.fail_with_status).