Repository navigation
fix(retain): resume a partially written oversized document slice by slice - #5676
Merged
Merged
Conversation
…lice A sliced retain that was interrupted part-way is retried with the same body. Slice 1 took the delta path, which diffs the complete body and extracts every unwritten chunk of the whole document in one step that commits only at the end: hundreds of LLM calls queued at once ahead of every other bank, and all of them lost if the retry is interrupted too. When the stored document already carries this body's hash but some chunks still differ, delta now steps aside. The streaming path resumes slice by slice: its recovery probe matches the same hash and skips the committed chunks. Fixes #5485
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5485
What was wrong
An oversized document is retained one slice at a time. If that retain was interrupted and then recovered, slice 1 took the delta path instead. Delta compares the whole body with what is stored and extracts every chunk not yet written in one step, saving nothing until the end. In the issue that was 463 and 716 LLM calls queued at once, which blocked other banks for over an hour. Every interrupted retry lost all of it.
Fix
In
_try_delta_retain: if the stored document already has this body's hash but some chunks still differ, it is a half-written copy of the same body. Delta now returnsNone. The normal streaming path then resumes slice by slice. Its recovery check matches the same hash and skips chunks already saved, so each slice is saved as it finishes.Real edits to a stored document are not affected (different hash). Neither is an identical re-retain (nothing to extract, so it stays on the metadata-only path).
Test
test_recovering_partially_written_oversized_document_resumes_slice_by_slice: crash a sliced retain after 3 extraction calls, then retain the same body again.calls: [17])Not in this PR
The other points in the issue still stand: a big edit to a stored document still runs as one unbounded delta step (option a), there is no fairness between operations in the shared LLM queue (d/e), and cancelling does not stop queued calls (f).