copy-db produces a silently corrupt, non-restorable copy and exits 0 — four independent channels in bin/copyDb.ts

[kriszyp]

copy-db produces a silently corrupt, non-restorable copy and exits 0 — four independent channels in bin/copyDb.ts

Summary

bin/copyDb.ts's copyDb() / copyDbi() — the pair behind both the documented copy-db <source> <target> CLI verb (bin/harper.ts:153) and the storage.compactOnStart flag — has four independent defects that each silently degrade the copy. The command exits 0 and logs copied N entries in every case.

Three were found by exploratory QA with execution-verified measurements; the fourth is a source-read finding in the same function. They are filed together because they live in one ~145-line function, share one root cause pattern (the copy loop was written as a raw byte-for-byte dbi walk and therefore cannot see any structure above raw key/value), and any fix touches the same code. They should be fixed as four changes, not one.

Severity order is roughly 1 > 2 > 3 > 4.

---

Channel 1 — the tombstone heuristic drops the shared-structures dictionary, blanking every record

bin/copyDb.ts:235:

// deleted entries should be 13 bytes long (8 for timestamp, 4 bytes for flags, 1 byte of the encoding of null)
if (value?.length < 14 && isPrimary) {
    skippedRecord++;
    continue;
}

This length heuristic also matches the table's shared-structures dictionary entry (stored under the Symbol.for('structures') key in the primary dbi). The dictionary is skipped as if it were a delete tombstone, so every record in the copy then decodes to null.

Unlike the RocksDB migration path in the same file (copyDbiToRocks, which explicitly does if (typeof recordKey === 'symbol') continue and separately calls copyStructures()), the LMDB copyDb() path has no symbol-key skip and no copyStructures() — it relies on the raw dbi walk to carry the symbol key through, and the heuristic eats it.

Measured through the CLI: 120/120 records readable before → 0/120 readable after. Raw keys all present (120/120), secondary index 5/120. So the rows are physically there and functionally blank.

Narrowings that must survive into the fix and its test:

• It is schema-shape-dependent — it fires only when the dictionary entry encodes to under 14 bytes, i.e. short attribute names. Realistic attribute names produce a larger dictionary and the table survives. That is why this has gone unnoticed.
• migrateOnStart / v4→v5 is not affected (different function, which handles structures explicitly).

Channel 2 — getKeys() + getEntry() collapses every dupSort secondary index

bin/copyDb.ts:230-233 walks the dbi with sourceDbi.getKeys({ start, transaction }) and then a single sourceDbi.getEntry(key, { transaction }) per key. On a dupSort dbi that yields one entry per unique key, discarding every duplicate.

Post-copy the secondary index therefore collapses to 1 entry per unique key.

Measured: groups of N=2, 10, and 100 duplicate values all collapse to 1. A unique-valued control index survives 112/112, which is what pins it to the dupSort path rather than to the copy loop generally.

Channel 3 — the blob store is never copied, so the copy is non-restorable

copyDb() walks LMDB dbis only. It never visits the blob store, which resources/blob.ts:1207 resolves as:

return [join(getHdbBasePath(), 'blobs', databaseName)];

— i.e. from the running instance's base path and the database name, not from the env file being copied. Each record's fileId reference is copied faithfully (the loop is a raw binary copy: dbiInit.encoding = 'binary', sourceDbi.decoder = null, so it never decodes a value and cannot know one holds a blob reference), while the bytes it points at are left behind.

Measured on 7863b7468: 6 blob docs @ 1.5 MB plus 4 sub-threshold inline controls. CLI exited 0 logging copied 11 entries. After swap-in: rows fully intact (10/10 readable — so this measurement is not confounded by Channel 1), 6/6 attachments threw BlobReadError 404, 4/4 inline controls byte-exact. Source blobs verified untouched.

Exposure differs by call site, and only incidentally:

• copy-db CLI — exposed. The target is an arbitrary path. Any copy that is later opened under a different database name, or moved to another data root or host, has lost every blob.
• compactOnStart — not currently exposed. It moves the copy back to the original dbPath under the same database name in the same base path (bin/copyDb.ts:84), so the untouched blob directory still resolves. Nothing enforces that invariant, though — it holds by accident of the call site, not by design.

Note the contrast with copyDbToRocks() in the same file, which does handle blobs deliberately (encodeBlobsWithFilePath, beginPendingMigrationBlobSaves, with comments citing #857 and #1337). The blob-awareness work landed on the migration path and never on the copy/compaction path.

Channel 4 — the audit store is copied into the source env, not the target, and isn't awaited

bin/copyDb.ts:216-220:

if (sourceAuditStore) {
    const targetAuditStore = rootStore.openDB(AUDIT_STORE_NAME, AUDIT_STORE_OPTIONS);
    console.log('copying audit log for', sourceDatabase, 'to', targetDatabasePath);
    copyDbi(sourceAuditStore, targetAuditStore, false, transaction);
}

rootStore is the source env (assigned from table.primaryStore.rootStore at line 171); the target is targetEnv, opened at line 177. So the variable named targetAuditStore is a handle on the source. lmdb-js's openDB constructs a fresh LMDBStore on every call (node_modules/lmdb/open.js:405-417 — no handle cache), so this is a new, live handle whose put was not covered by the deliberate write-guard at lines 160-170 (table.primaryStore.put = noop etc.).

Two consequences:

1. The target copy gets no audit / transaction log at all.
2. The copy loop writes source audit entries back into the source database during an operation that is documented and guarded as read-only on the source. The re-put is key-for-key identical so the data damage is likely nil, but it dirties the source env mid-compaction and defeats the write-guard's purpose.

Separately, this copyDbi call is not awaited — compare line 214, await copyDbi(sourceDbi, targetDbi, isPrimary, transaction), for the regular dbis. The un-awaited writes race transaction.done() and targetEnv.close() in the finally at lines 293-296.

Channel 4 is source-read only — I have not run it. Channels 1–3 carry execution-verified numbers.

---

Suggested direction

Four separate changes:

1. Channel 1 — skip symbol keys explicitly and copy the structures entry explicitly, the way copyDbToRocks already does, instead of inferring "tombstone" from a length. A 13-byte length test cannot distinguish a tombstone from a small dictionary and shouldn't be asked to.
2. Channel 2 — walk dupSort dbis with getRange/getValues so duplicates are preserved.
3. Channel 3 — decide the contract. Either copy-db copies <hdbBasePath>/blobs/<databaseName> alongside the dbis, or it refuses to run against a database that has blob attributes, or it prints a loud warning naming the directory the operator has to copy themselves. Silently emitting a copy that cannot be restored is the one option to rule out. Whatever is chosen, compactOnStart's same-name/same-root assumption should become an assertion rather than a coincidence.
4. Channel 4 — targetEnv.openDB(...), and await the call.

Verification anchors

Channels 1–3 have QA specs with the measurements quoted above; they should land as regression tests with the fixes. Channel 1's test must use short attribute names, since realistic ones don't reproduce it.

Provenance

Found by the qa-explorer exploratory QA loop (findings F-180, F-178, F-184). Channel 4 found on source review while consolidating them. Source lines cited against main @ b852a722c.