[nicoloboschi]

Problem

Concurrent retains targeting the same document have two issues, both reproduced empirically (see tests/test_retain_same_document_concurrency.py):

1. Redundant LLM extraction in the delta hash-mismatch race. When a concurrent retain commits a new version of a document between our chunk-load and our write, the delta path falls back to a full re-process — but the expensive LLM extraction (_extract_and_embed) has already run by then, because it happens before the post-load hash check. Acting only after the spend can't recover the tokens.

2. Postgres deadlock in the streaming fallback. Concurrent same-document retains with non-overlapping content deadlock (DeadlockDetectedError, 3-way lock cycle in handle_document_tracking). The ownership gate used INSERT ... ON CONFLICT DO NOTHING (which does not lock the existing row) followed by a separate SELECT ... FOR UPDATE, letting writers interleave the speculative-insert ShareLock, the FOR UPDATE, and the cascade-DELETE in inconsistent orders.

Context: this came out of investigating #1916. That PR tries to recover after the hash check, which (a) can't save the already-spent extraction, and (b) crashes every racing delta retain with NameError: _recovery_no_op (nested-scope variable read in the outer scope). Details in the PR #1916 thread.

Fix

1. Pre-extraction freshness recheck (_try_delta_retain): re-read the document hash immediately before extraction. If a concurrent retain already committed identical content, skip extraction and update metadata only; if it still differs, fall back to streaming. This acts before the LLM spend.

2. Atomic ownership gate (_streaming_retain_batch): replace INSERT ON CONFLICT DO NOTHING + SELECT FOR UPDATE with a single INSERT ... ON CONFLICT DO UPDATE SET content_hash = documents.content_hash RETURNING content_hash. DO UPDATE takes the row lock atomically, so all writers serialize on the document row in one consistent step.

3. Extracted the chunk-diff classification into _classify_chunk_diff returning a typed _ChunkDiff (no tuple return), reused by both the main diff and the recheck.

Verification

tests/test_retain_same_document_concurrency.py (counts real fact_extraction.extract_facts_from_contents calls; stub embeddings because the local torch model segfaults under in-process async concurrency):

Scenario	Before	After
10 concurrent identical retains	0 extractions	0 extractions (guarded)
10 staggered partial-overlap retains	~10 extractions	1 extraction
10 concurrent fully-different retains	deadlock	no deadlock, all succeed
10 concurrent partial-overlap (burst)	10	10, crash-free

Note: a tight simultaneous burst still extracts N times — the recheck can't see commits that haven't happened yet. Fully eliminating that would require per-document serialization (advisory lock), intentionally out of scope here.

All existing delta/retain suites pass; lint + ty clean.

Risk

Low. The recheck adds one indexed PK lookup to delta retains that have changes and otherwise preserves behavior (the post-extraction hash gate remains the correctness backstop). The ownership-gate change is a strict tightening of locking on a path that already used ON CONFLICT.