Skip to content

Document copy-db's blob companion directory and restore steps #620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kriszyp wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Document copy-db's blob companion directory and restore steps #620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
73b8c2f
Document that copy-db copies file-backed blobs beside the target
kriszyp Aug 6, 2026
5d01608
Address review feedback on copy-db blob docs
kriszyp Aug 26, 2026
8efb794
Correct blob copy claims from pre-push review
kriszyp Aug 26, 2026
d25adaf
Say precisely what copy-db does with a RocksDB database
Aug 26, 2026
d4ba557
Sharpen blob-root paths and multi-root restore steps
Aug 26, 2026
f69fe91
Scope the companion-directory rule and note the compact-on-start backup
Aug 26, 2026
3a2ba6a
Point LMDB users at an LMDB backup path, flag the stricter target rule
Aug 26, 2026
5caf515
Note that LMDB get_backup streams the database file only
Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions reference/cli/commands.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,8 @@ This command regenerates the self-signed SSL/TLS certificates used by Harper.

Available since: v4.1.0

<VersionBadge type="changed" version="v5.3.0" />

Copy a Harper database with compaction to eliminate free-space and fragmentation.

```bash
Expand All @@ -250,8 +252,8 @@ harper copy-db <source-database> <target-database-path>

**Parameters**:

- `<source-database>` - Name of the source database
- `<target-database-path>` - Full path to the target database file
- `<source-database>` - Name of the source database (a name, not a file path)
- `<target-database-path>` - Full path to the target database file; neither it nor its `<target-database-path>-blobs` companion directory may already exist. Retrying an interrupted copy means removing both.

**Example**:

Expand All @@ -261,6 +263,8 @@ harper copy-db data /home/user/hdb/database/copy.mdb

This copies the default `data` database to a new location with compaction applied.

As of v5.3.0 the database's `Blob` files are copied to `<target-database-path>-blobs/<rootIndex>/`, since blob files live outside the database file and are addressed by database name. If the database holds `Blob` values, the copy is not restorable without that companion directory — see [Database Compaction](../database/compaction.md#file-backed-blobs-copied-separately) for the restore steps. LMDB databases only — `copy-db` fails if the source database is stored in RocksDB, which compacts itself.

**Use Cases**:

- Database optimization
Expand Down
43 changes: 42 additions & 1 deletion reference/database/compaction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ Compaction is also the mechanism to apply storage configuration changes (such as

## Copy Compaction

<VersionBadge type="changed" version="v5.3.0" />

Creates a compacted copy of a database file. The original database is left unchanged.

> **Recommendation:** Stop Harper before performing copy compaction to prevent any record loss during the copy operation.
Expand All @@ -29,18 +31,57 @@ harper copy-db <source-database> <target-database-path>

The `source-database` is the database name (not a file path). The target is the full file path where the compacted copy will be written.

To replace the original database with the compacted copy, move or rename the output file to the original database path after Harper is stopped.
As of v5.3.0 neither the target path nor its `<target-database-path>-blobs` companion directory may already exist — `copy-db` refuses both rather than merging the copy into whatever they hold. Retrying an interrupted copy means removing both. This is stricter than earlier v5 releases, which wrote into an existing target: a script that re-copies to a fixed path on a schedule has to remove the previous copy and its companion directory first, or it now fails.

To replace the original database with the compacted copy, move or rename the output file to the original database path after Harper is stopped. That is the one case where the database file travels alone; if the database has `Blob` values, any other destination also needs the blob companion directory described in [File-backed blobs copied separately](#file-backed-blobs-copied-separately).

**Example — compact the default `data` database:**

```bash
harper copy-db data /home/user/hdb/database/copy.mdb
```

Copy compaction applies to LMDB databases only. `copy-db` fails if the source database is stored in RocksDB, and [compact on start](#compact-on-start) skips RocksDB databases — RocksDB compacts itself.

### File-backed blobs copied separately

<VersionBadge type="changed" version="v5.3.0" />

`copy-db` copies the database's blob files alongside the copy. Earlier v5 releases copied only the database file, leaving the blobs behind.

`Blob` values are stored outside the database file (unlike `Bytes` values, which are stored inside the record). These blob files live in the configured blob roots — `<storage.blobPaths[n]>/<database>`, or `<rootPath>/blobs/<database>` when `blobPaths` is not configured — and are addressed by **database name**, not by the path of the database file.

`copy-db` therefore writes them alongside the copy:

```
<target-database-path>-blobs/<rootIndex>/...
```

`<rootIndex>` is the position of the source root in the database's blob-root list, preserved so a multi-root database restores each root to its original slot. A `README.md` in that directory records the mapping.

**If the database holds `Blob` values, the copy is not restorable without this directory** — moving the database file on its own silently loses every blob it references. A database with no live blob references does not need the companion directory, though one may still be written (possibly empty) whenever a blob root directory exists.

The directory is written only for blob roots that exist on disk, and is not written at all when every root is missing — an unmounted `blobPaths` volume, for instance, yields a database-file-only copy without failing the command. Confirm the roots are mounted before copying, and confirm `<target-database-path>-blobs` is there afterwards, before treating the copy as a restorable backup.

To restore the copy under a database name, put each `<rootIndex>` tree into that name's matching blob root — for example, restoring the copy above as a database named `archive` with no `storage.blobPaths` configured:

```bash
cp /home/user/hdb/database/copy.mdb /home/user/hdb/database/archive.mdb
cp -r /home/user/hdb/database/copy.mdb-blobs/0/. /home/user/hdb/blobs/archive/
```

A database with several `storage.blobPaths` entries has one `<rootIndex>` tree per root: restore every one of them into the slot of the same index. Leaving a tree behind loses exactly the blobs that lived on that root, and neither the copy nor Harper reports it.

One narrow exception: if the copy immediately replaces its own source in place — same installation, same database name, before anything writes to the source — the database file alone is enough, since the blob roots it references are still exactly as the copy left them. A copy kept as a backup does not qualify: once the source is written to, Harper can reclaim the blob files an older copy still references, so restore the companion directory along with the database file.

## Compact on Start

Automatically compacts all non-system databases when Harper starts. Harper will not start until compaction is complete. Under the hood, it loops through all user databases, creates a backup of each, compacts it, replaces the original with the compacted copy, and removes the backup.

Compact on start replaces each database in place under its own name, so the blob roots keep resolving and no blob companion directory is involved. As of v5.3.0 it skips RocksDB databases, and skips a database whose tables span more than one storage environment (table-specific paths), which compaction cannot replace as a single file.

> **Note:** the backup `compactOnStartKeepBackup` retains is the pre-compaction database file only. It carries no blobs, and blob files are shared by database name, so blobs deleted or superseded after the compaction are gone from that backup's point of view. Treat it as a rollback for the compaction itself, not as a point-in-time backup. For that, an LMDB database needs a volume snapshot covering the database file and its blob roots together, or a `copy-db` copy kept with its blob companion directory — [`get_backup`](../backups/operations.md#get_backup) on an LMDB database streams the `.mdb` file only.

Configure in `harper-config.yaml`:

```yaml
Expand Down