Add RocksDB managed backup and restore operations

[cb1kenobi]

Summary

Adds managed backup/restore for RocksDB databases as Operations API operations — create_backup, list_backups, verify_backup, delete_backup, purge_backups, restore_backup — plus RocksDB support for get_backup (streams a gzipped-by-default tar; gzip=false for plain tar; works against a remote target=). Each is also runnable from the CLI under its operation name, and offline against a stopped server. Managed backups are an incremental, checksum-verified, server-side repository under storage.backupPath (default <rootPath>/backup), one subdirectory per database, capturing the transaction log alongside the data.

Purpose

Give RocksDB first-class server-managed backup/restore (LMDB already had volume-snapshot workflows), without stopping the server for the common case.

Where to focus review

• Restore safety invariant — dataLayer/rocksdbBackup.ts (restoreBackup) + dataLayer/restoreMarker.ts. Online restore writes an fsynced marker + takes a per-db flock, closes the DB across all worker threads, verifies process-wide closure via registryStatus(), then purges + rewrites. A component-held DB (and always system) can't be closed → 409 pointing at offline restore. Worth checking the close→verify→purge ordering and the marker/lock crash-recovery states (in-progress / incomplete / clear).
• Offline restore lock probe — restoreBackupOffline. The offline path is chosen on a PID heuristic, but the PID file is briefly absent during harper restart, so it now probes RocksDB's own lock (RocksDatabase.open) before purging — a lock error aborts, a corrupt/half-restored dir still proceeds (recovery).
• Job-worker handle release — server/jobs/jobProcess.ts + resources/databases.ts (closeDatabase/closeLoadedDatabases, new). Job workers close their process-global RocksDB handles on exit or the closure check can never pass; system is intentionally left open; tableless databases are handled explicitly.
• CLI transport refactor — bin/cliOperations.ts. Extracted target + auth resolution into resolveRequestOptions, now shared by cliOperations and the streaming get_backup download. cliOperations behavior should be unchanged (bin suite green).

Cross-model review (codex + Gemini)

Fixed from review: offline-restore lock probe, CLI content-disposition path traversal, jobProcess cleanup when updateJob rejects, marker-removal fsync, tableless-DB handle leak. Deferred (not blocking):

• Enforcing "DB closed" via a native atomic RocksDB lock instead of registryStatus() polling — the cross-process case is covered by the offline probe; a native primitive is a rocksdb-js follow-on.
• Table background-work disposal (expiration timers/callbacks) on close — pre-existing (resetDatabases shares the gap), filed as Table background work (expiration timer, delete/cleanup callbacks) not disposed when a database's stores are closed #1811.

Docs

Companion docs PR: HarperFast/documentation#590.

Generated by an LLM (Claude Opus 4.8, 1M context).

🤖 Generated with Claude Code

---

Update — blob backups + restore-safety review (2026-07-29)

This revision addresses the review feedback (kriszyp / GPT 5.6). Merged latest main first (one conflict in bin/cliOperations.ts, resolved).

Blobs (exclude_blobs, default false → blobs included)

File-backed blobs live in roots outside the RocksDB directory, so the engine backup misses them. New dataLayer/blobBackup.ts captures them alongside the engine data:

• Managed backups snapshot each blob root to <backupDir>/blobs/<backupId>/<rootIndex>/<relpath> — a full per-id copy (like the binding's transaction-log snapshots), hard-linked when possible, copied across filesystems, never symlinked, built in a temp dir and atomically renamed. restore_backup purges each root and rewrites it from the snapshot; delete_backup/purge_backups remove the matching snapshots. Wired for both the online ops and the offline CLI.
• get_backup appends blob files to the same tar under blobs/<rootIndex>/…. Since the binding finalizes its tar with a fixed 1024-byte end-of-archive marker, createBackupStream streams the native plain tar minus that trailer, appends blobs via tar-stream, and gzips the combined stream itself.
• get_backup CLI help updated.

Restore-safety fixes

• Offline lock probe (was the data-loss bug): now recognizes rocksdb-js 2.5.0's real LOCK-file error (a plain Error, no code, "While lock file: …/LOCK: Resource temporarily unavailable"), takes the restore lock+marker before probing, and fails closed. Two-process regression test included.
• Restore-metadata namespace: moved from <db>.restoring/<db>.restore.lock siblings to an isolated .restore/ dir keyed by a hash of the db-dir name — fixes the collision with a legal database named orders.restoring and the NAME_MAX overflow on 250-char names.
• Recovery marker: beginRestore reports a pre-existing marker so a failed recovery attempt no longer clears a marker over a possibly half-purged directory.
• Drop vs restore: dropDatabase now holds the per-db restore lock across the whole drop (serializing with restore on one primitive) instead of a check-then-act marker probe.

Please look at

• bin/cliOperations.ts auth: the merge resolution moved credential resolution into resolveRequestOptions, which ran before cliOperations' try/catch — so an incomplete auth_username=/auth_password= pair escaped the exit(1) handling. Moved back inside the try; the 4 harper#1872 credential tests pass again. Worth a sanity check that CLI auth precedence is unchanged.
• get_backup tar merge depends on the binding's fixed 1024-byte trailer; the trailer-strip verifies the withheld bytes are all-zero and throws loudly if the format ever changes.
• Blob snapshot consistency is best-effort point-in-time (whatever files exist while it streams; a blob deleted mid-walk is skipped) — Harper does not freeze blob writes for a backup, the same tradeoff the engine makes for the transaction log. Hard-linking is safe because blobs are content-addressed/write-once (see blobBackup.ts header).
• Behavior change: dropDatabase now refuses (409) a database with an incomplete-restore marker, same as it already did for an in-progress restore.

Unit suites green (dataLayer, resources, bin); relying on CI for the integration suite. New tests: blobBackup.test.js, extended rocksdbBackup.test.js (blobs, exclude_blobs, get_backup tar contents, two-process lock probe), rewritten restoreMarker.test.js (collision/NAME_MAX/preexisting/scan), databases.test.js (drop-vs-restore), bin/backup.test.js (atomic download).

Generated by an LLM (Claude Opus 4.8, 1M context).

[kriszyp]

Looks awesome! GPT mostly has concerns about the lock handling and timing with failures. But probably the biggest gap is blob file handling. We could independently handle blob files, but kinda makes the backup system a bit incomplete. We might want to consider some way of creating a snapshot directory that adds hard-links to the blob files so they can exist as snapshots alongside the rocksdb backups? (And include them in the tar'ed snapshot for get_backup). I'd also be fine with blob backup being a follow-up PR and saying backups don't support blobs for now. Good work!

_{🤖 Reviewed with GPT 5.6}

[kriszyp]

This looks like a solid mechanism, I like this, good work. I think the codex findings are reasonable requests. The snapshot consistency is certainly the most interesting/challenging, but I think the suggestion of having a mechanism to delay/pause blob deletions during the backup process makes sense to me.
_{🤖 Reviewed with GPT 5.6}

[kriszyp]

I'll drop this in for the comments since it is still marked ready, but I assume this is still a bit of WIP with the blob/database coordination.
_{🤖 Reviewed with GPT 5.6}

[kriszyp]

Looks good, nice work. I'm including the comments; codex thinks they are blockers/major, but I think we've agreed it is reasonable follow-up work, so I'm good with merging this.
_{🤖 Reviewed with GPT 5.6}

[kriszyp]

The binding's backup-directory writer lock ends when rootStore.backup() resolves, before finalizeBackup() copies blobs and publishes the manifest. A concurrent purge_backups can therefore acquire that lock, see and purge this new engine backup, and return while this path continues copying blobs, writes a manifest, and reports a successful backup_id that no longer exists in the engine. Please hold one lifecycle lock across engine creation, blob capture, and manifest publication (and use that same lock for delete/purge), or add an equivalent final engine-presence check under serialization with rollback. A regression can pause snapshotBlobs, run purge, then assert creation cannot return success for an absent backup.

[cb1kenobi]

Real race — the engine backup lock releases before finalizeBackup copies blobs + writes the manifest, so a concurrent purge can delete the engine backup while create finishes. Tracked as a follow-up to serialize backup management ops under a per-database lock: #2031.

[kriszyp]

This skip prevents startup from opening the reserved directory, but it still leaves the collision case unresolved. If the reserved `restore` path is already a RocksDB (possible because discovery historically accepted CURRENT + MANIFEST-* directories without applying schemaRegex), an upgrade now silently makes that database disappear, and the next restore writes its .lock/.restoring files into the same directory via acquireRestoreLock. Please detect an existing non-metadata directory before treating this path as lifecycle state and either refuse startup/restore with an actionable error or migrate the metadata outside the database root. The regression should use a real RocksDatabase containing a row and verify its data is not merely skipped.

[cb1kenobi]

Agreed — the scan skip avoids loading it, but a pre-existing on-disk `restore` database (only reachable via manual creation) would silently disappear. Tracked as a follow-up to warn/refuse instead of silently skipping: #2033.

[kriszyp]

A loopback last_target does not establish that the target is this Harper installation: it may be an SSH tunnel, a container-published port, or another local Harper root. If that endpoint is currently down and this installation's PID is absent, restore_backup/delete_backup/purge_backups now silently operate on this installation's files instead of the saved target. Please make offline mode explicit or persist an unambiguous local-installation identity rather than inferring it from the hostname. Add a regression with a saved loopback target for a different port/installation and assert that it never falls through to direct file access.

[cb1kenobi]

Fair — a loopback last_target may be a tunnel/container/another root, so silently routing destructive offline ops to local files is a footgun. Tracked as a follow-up (restrict local fallback to read-only, or gate destructive ops behind an explicit offline flag): #2032.