[cdbartholomew]

Problem

RetainResult.processed_content_tokens tells a post-retain hook how much content actually went through extraction after chunk-level dedup:

• None: no dedup signal; assume the full submission was processed.
• 0: nothing changed.
• N: only N tokens were extracted.

For re-retains of oversized documents it is wrong:

1. An item over retain_batch_tokens (default 10,000) is split into sub-batches by the async retain path.
2. Only the first sub-batch may take the delta path (attempts_delta_retain).
3. The later sub-batches run _streaming_retain_batch. There they find their chunks already committed (the per-chunk recovery hashes) and skip them, but that function returned processed_content_tokens=None unconditionally.
4. None is contagious across sub-batches (merge_processed_content_tokens), so the whole retain reports "no dedup signal".

The result is that a large document re-submitted with little or no change (for example, a growing transcript upserted under a stable document_id) is reported exactly like a full first retain, even though almost nothing is re-extracted. Any extension that meters by this field counts the whole document again on every re-retain.

Fix

_streaming_retain_batch's producer already decides, per chunk, whether to skip it (hash committed) or send it to extraction. In recovery mode — the document row's content_hash matched and chunks are already committed, so existing_chunk_hashes is non-empty — it now also counts the content + context tokens of the chunks it sends, the same quantity _count_delta_content_tokens reports on the delta path, and returns that instead of None.

Outside recovery there is no per-chunk dedup at all, so the return stays None — "bill the submitted payload" — which is also what every path that never reaches the producer keeps doing.

The condition is recovery rather than "at least one chunk was skipped": a sub-batch whose chunks all differ from the committed ones (the appended tail of a grown document) did extract all of them, and saying so is accurate, where one None would re-inflate the whole retain back to the full submission. Counting is skipped entirely outside recovery, so a first retain does not tokenize the body a second time for a number that is thrown away.

No change to what is extracted, chunked or stored; only the reported number changes.

One behavioural note: a retain resumed after a crash also skips the chunks its earlier attempt committed. It now reports only the remainder, consistent with what the delta path already reports for unchanged chunks.

Tests

Two tests added to tests/test_delta_oversized_replacement.py:

• Unchanged re-retain: an oversized document re-retained unchanged reports 0. Before this change it reported None with zero LLM input tokens.
• Edited re-retain: an oversized re-retain with one edited section and an appended tail reports no less than what reached extraction, and well under the submission.

Both fail without the fix and pass with it.

The _RetainResultCapture validator these need was already copy-pasted in test_delta_retain.py and test_document_transfer.py; it now lives once in tests/retain_result_capture.py and all three import it.

https://claude.ai/code/session_011Fo4NboRJPywxb1UfwpneJ