Skip to content

fix(backup): include all 7 missing tables in backup/restore - #1903

Merged
nicoloboschi merged 1 commit into
mainfrom
fix/backup-restore-missing-tables
Jun 1, 2026
Merged

nicoloboschi merged 1 commit into
mainfrom
fix/backup-restore-missing-tables

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Problem

BACKUP_TABLES in hindsight-api-slim/hindsight_api/admin/cli.py listed only 8 tables, but the live PostgreSQL schema has 15 (verified by building the schema on embedded pg0 and introspecting information_schema).

7 tables were never backed up: mental_models, directives, async_operations, webhooks, file_storage, audit_log, graph_maintenance_queue.

This was worse than missing data. Restore runs TRUNCATE banks CASCADE, and mental_models, directives, async_operations, webhooks all have a FK to banks — so restore was actively wiping those tables (including core user data like mental models and directives) and never refilling them.

Fix

  • admin/cli.py — add the 7 missing tables to BACKUP_TABLES in FK-dependency order (verified against the live schema's actual foreign keys), with a comment explaining the completeness requirement.
  • tests/test_admin_backup_restore.py
    • test_backup_tables_covers_entire_schema — introspects the live schema and asserts BACKUP_TABLES equals the real table set (minus alembic_version). Fails CI if a future migration adds a table without updating the list.
    • Roundtrip test now inserts a directives row (FK→banks) and asserts it survives restore, directly covering the cascade-wipe regression.
  • .claude/skills/code-review/SKILL.md — new review step + "must fix" bullet flagging any new PG table missing from BACKUP_TABLES.

Notes

  • observation_sources is intentionally excluded — it's Oracle-only (PG uses an array column); admin backup/restore is PG-only.
  • Transient queue tables (async_operations, graph_maintenance_queue) and audit_log are included so a backup is a faithful full snapshot. If we'd rather a restore not carry over another deployment's in-flight worker queue, these can move to an explicit documented exclusion set instead.

Testing

All 6 tests in test_admin_backup_restore.py pass; lint + format clean.

BACKUP_TABLES listed only 8 of the 15 live PostgreSQL tables. The 7
missing tables (mental_models, directives, async_operations, webhooks,
file_storage, audit_log, graph_maintenance_queue) were never backed up,
and because restore runs TRUNCATE banks CASCADE, the FK-to-banks children
(mental_models, directives, async_operations, webhooks) were actively
wiped on restore even though they were never saved.

Add the missing tables in FK-dependency order, plus a guard test
(test_backup_tables_covers_entire_schema) that introspects the live
schema and fails if BACKUP_TABLES drifts from it. Extend the roundtrip
test with a directive (FK->banks) to cover the cascade-wipe regression.

Document the rule in the code-review skill so new tables don't silently
escape the backup list.
@nicoloboschi
nicoloboschi merged commit 867b7b4 into main Jun 1, 2026
76 of 77 checks passed
nicoloboschi pushed a commit that referenced this pull request Jun 5, 2026
#1903 expanded BACKUP_TABLES to all 15 tables (the 7 previously-missing ones that could be silently dropped on restore), but the "backup includes" list still reflected the old ~8-table coverage. Update it to match: mental models, directives, webhooks, file storage, plus internal operational tables for a faithful full-database snapshot. Oracle-only observation_sources stays excluded (PostgreSQL-only backup). Regenerated skills/hindsight-docs mirror.
cdbartholomew added a commit that referenced this pull request Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant