Base-copy replication wedges in `Receiving` when sender has records referencing blob files missing on disk

[ldt1996]

Summary

When a v5 node attempts a base-copy resync to a peer, and the sender's primary store contains any record whose Blob field references a file that doesn't exist on the sender's disk, the receiver's connection wedges permanently in lastReceivedStatus: "Receiving". Base-copy never reaches COPY_COMPLETE, the watermark never advances, the blob-bearing database's replication is fully stuck. No errors at the API level; symptom appears only as silent record propagation loss between specific peer-pairs.

Minimal reproduction

1. Start one v5 node, deploy a fixture with a Blob field, insert N records with content above FILE_STORAGE_THRESHOLD (8192 bytes) so blobs go to disk files
2. Delete a contiguous block of blob files directly from <dataRoot>/blobs/<db>/
3. Add a fresh v5 peer; receiver requests base-copy resync
4. Observe cluster_status.connections[0].database_sockets[].lastReceivedStatus stays at "Receiving" indefinitely; only records before the gap apply

[blob-gap] source has 60 blob files
[blob-gap] deleted 40 contiguous blob files (simulating v4 gap)
[blob-gap] t=0s receiver status=Receiving record_count=7/60
[blob-gap] t=85s receiver status=Receiving record_count=7/60

Migration is NOT required for reproduction — the bug fires on any v5 source whose primary store references a missing blob file. We initially suspected the v4→v5 migration path but ruled it out: migration logs are clean, and the reproduction does not use a v4 binary.

Test results

State	Receiver status after ~60s	Record count
Pre-fix	Receiving (stuck)	7/60

Test asserts post-fix behavior. Same setup, same parameters, deterministic.

Cross-references

• harper-pro#352 — separate transient shared-structure misread surface at boot.

[kriszyp]

Diagnosis: the trigger is application code calling blob.save(), which isn't a Harper API — not a core defect.

Pulled the logs from the affected cluster (cluster:v4-to-v5 customer:akamai). The whole thing is one base-copy resync window (2026-06-17 00:47–02:37 UTC), and the order of events is:

1. TypeError: blob.save is not a function — 1,266×, on se-sto, thrown while resolving PageCache records (the cache source-resolution path in Table.ts), immediately before the timeout storm.
2. Then Timeout waiting for blob stream … — 242,257×, including an uncaughtException variant.

blob.save() is not a method on Harper's blob type — it doesn't exist anywhere in core / harper-pro / legacy harperdb (only test files reference it). So this is application/component code (the PageCache resolver) calling a non-existent method on a blob it gets back from a v4→v5-migrated record. That throw aborts the record's apply before the blob's save stream is wired up, which is what leaves the receiver unable to finish those blobs and the base-copy unable to converge.

So the record loss here is downstream of an application bug. The fix is for that component to stop calling blob.save() — blobs are persisted as part of the normal record write, there's no explicit save to call. Nothing for Harper to change to make the trigger go away.

What we are hardening (#409): the orphaned blob stream that the failed apply leaves behind was being torn down by the blob-timeout sweep with no 'error' listener, which Node promoted to a process-level uncaughtException — that's the 242k storm, and it can destabilize the node well beyond the one bad record. #409 makes the receiver absorb that teardown so a blob that errors / never saves (for any reason, app bug included) degrades gracefully instead of crashing the process. It deliberately does not make the copy converge while the app keeps calling blob.save().

— Claude (Opus 4.8)