Torn audit entry throws RangeError out of transaction-log iteration instead of degrading

[heskew]

Summary

A torn audit entry causes readAuditEntry to throw a RangeError out of transaction-log iteration, surfacing as an unhandled rejection rather than being handled as recognizable corruption.

Observed on 5.1.x, on both the main thread and a worker:

Reading audit entry error RangeError: Audit entry username length 436207616
  exceeds remaining buffer (position 116, byteLength 117)
    at readAuditEntry (.../core/resources/auditStore.js:543)
    at ExtendedIterable.<anonymous> (.../core/resources/RocksTransactionLogStore.js:436)

A declared username length of 436,207,616 against a 117-byte buffer is not an ambiguous edge case — it is a corrupt record. Throwing from inside the iterator means the corruption takes out whatever was iterating.

How the record got torn

The instance had exhausted its filesystem quota, so writes were being refused mid-operation. That left a partially written entry, which persisted across the restart. Any abrupt write failure — full disk, quota, power loss, crash — can produce the same tear.

The replay path already handles this correctly

The replication replay path degrades gracefully on the analogous condition, at warn level, and keeps the process healthy:

[warn]: Stopping transaction log "local" at a corrupt entry during replay
  RangeError: Corrupt transaction log entry at position ...:
  declared length 1688494450 overruns the log (limit=...)

So there are two readers of potentially torn records and only one of them degrades gracefully. The audit reader should behave like the replay reader.

Suggested direction

Treat an implausible declared length as corruption at the read boundary: log it, and stop or skip cleanly the way replay does, rather than propagating a RangeError to the caller. A bounds check against the remaining buffer before trusting the length field is enough to distinguish "corrupt record" from "programming error," which is the distinction the current code can't make.

Worth deciding as part of this whether a corrupt entry should stop iteration (replay's choice, safest for ordering guarantees) or skip and continue — they have different implications for audit completeness, and the answer may differ from replay's.

---

🤖 Filed by Claude on behalf of @heskew

[kriszyp]

The stack in the issue is emitted from the existing readAuditEntry() catch block, not evidence that the RangeError escaped it. Both v5.1.22 and v5.1.24 contain the handling from #577: the length guard throws inside readAuditEntry()’s try, the catch logs Reading audit entry error, and then returns a skip-eligible sentinel (tableId/type undefined). RocksTransactionLogStore.getRange() also wraps the per-entry decode.

rocksdb-js#723 operates below this layer. It recovers the committed watermark and discards a provably unclosed transaction tail. That can eliminate one narrow source of malformed trailing data, but it does not validate Harper audit payload fields or guarantee graceful handling of a structurally framed corrupt payload, so it should not be treated as fixing this issue.

To identify a remaining defect, we need the separate error after this log: the unhandledRejection/uncaughtException stack, which consumer was iterating (replication/subscription/audit query), and whether iteration stopped or continued. Ideally we should also capture the transaction-log name and entry position. Without that second stack, this issue appears either already handled by #577 or is an observability problem (caught corruption logged at error level without enough position/context), rather than an escaping RangeError.

[heskew]

Escalation on the same tear family: #2016 documents a confirmed incident where the other reader of this class of torn entry — crash-recovery replay via endIteratorOnCorruptFrame — stopped at a mid-log corrupt frame and silently rolled a table back 2.2 days, dropping 33 acknowledged records (recovered only from an off-box backup). Replication readers stopped at the same frame, starving the peer for the same period with green status throughout.

Relevance here: the open question in this issue ("stop iteration vs skip and continue") now has incident data. Stopping is only safe when the tear is at the tail; on a mid-log tear it amputates valid acknowledged entries. Whatever this issue decides for the audit reader should be consistent with the resolution of #2016 — and "throw a RangeError at the caller" (current behavior here) is still the worst of the three options.

---

🤖 Posted by Claude on Nathan's behalf

[kriszyp]

Exploratory QA has hit this four independent times on the Bun runtime, where it has a second-order symptom worth recording here rather than in a new issue: the process dies ~3 seconds after the operation reports COMPLETE.

Shape of the Bun reproduction:

1. delete_transaction_logs_before returns COMPLETE.
2. A RangeError is thrown while reporting success — a JS-level decode failure over the same torn/misaligned audit entries this issue describes.
3. The process exits ~3s later with no shutdown line in the log.

The earlier hypothesis was a native-addon abort. The RangeError rules that out and points at exactly the mechanism described here: a RangeError escaping the prune's async iteration becomes an unhandled rejection, which on Bun takes the process down silently rather than logging. That would explain the error and the delayed death with a single cause, and it makes the fix proposed in this issue — degrade at the read boundary instead of throwing out of the iterator — the fix for the Bun crash as well.

Two things this adds to the decision already framed above:

• The choice between "stop iteration" and "skip and continue" now has a second consumer. The prune path is destructive, so stopping at a corrupt entry leaves the operator with a partial purge that reported COMPLETE. Whatever is chosen, the prune's reported result needs to reflect that it stopped early — reporting COMPLETE and then dying is the worst combination available.
• The unhandled-rejection path deserves a guard of its own. Even with a bounds check at the read boundary, an exception escaping the prune's async iteration into an unhandled rejection is a process-death mechanism on Bun that logs nothing. That's independently worth closing, since it will silently swallow the next unrelated throw on that path too.

Not yet verified: whether the RangeError → unhandled rejection → exit chain is actually what kills the process, or whether the delayed death has another cause. It's a one-run experiment (attach an unhandledRejection handler and see whether the exit becomes a logged error) and nobody has done it.

Also unresolved: whether this reproduces on the other Bun operation arms — 8 of 9 are unrun — so class-vs-single-operation is open. This currently blocks the Bun shard of #1916.

_{Reported by the qa-explorer exploratory QA loop (finding F-264). Filed as a comment here rather than a new issue: same root cause as this issue, different surface.}