Skip to content

fix(transfer): import an archive without holding it in memory (#5524) - #5696

Merged
nicoloboschi merged 2 commits into
mainfrom
fix/import-streaming-5524
Oct 16, 2026
Merged

nicoloboschi merged 2 commits into
mainfrom
fix/import-streaming-5524

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Follow-up to #5661. Closes #5524.

Import used to hold a bank in memory several times over: the whole uploaded zip, every parsed document, every observation, every document's text until the end (for the metering hook), and one remapped id per fact and observation. Now it reads the archive as it replays it.

What changed

  • Archive stays on disk. The worker copies the stashed archive into a temp file (retrieve_stream) and reads zip entries from there. The upload endpoints pass the spooled upload file instead of await file.read(); submission stores it with store_stream.
  • Documents are parsed one entry at a time and batched as before. Conflict ids are resolved per batch.
  • Observations and history rows (observations.json, history/audit_log.json, history/llm_requests.json) are decoded one array item at a time by a small incremental JSON reader, so json.loads no longer builds the whole list.
  • Attachment bytes are read from the zip when each one is written, not all up front.
  • Metering hook: document text is no longer kept for every document until the end. After the import, the hook re-reads each document from the archive (only when a validator is configured).
  • Id maps: the per-fact (doc, ordinal) -> unit id map is now one list per document. remapped_unit_ids keeps only the ids that mental models, their history, the graph queue and the curation archive actually cite, instead of every fact and observation.
  • Fewer round trips: each observation batch writes created_at, event_date and source links with one executemany per statement, not up to 3 queries per observation. _restore_rows writes 500 rows per executemany.
  • Submission reads only the manifest for a bank restore (plus the documents when a validator gate runs), instead of parsing the whole bank archive.

Measured

Synthetic archive with stubbed embeddings, peak Python memory (tracemalloc) for the whole import:

10k facts + 10k observations 20k facts + 20k observations
main 114 MB timed out at 300s
this PR 25 MB 25 MB

Peak memory now stays almost flat as the bank grows. Speed is about the same; tracemalloc makes the timings noisy.

What still grows with the bank

  • One new unit id per fact (needed to resolve observation sources): roughly 90 MB per 1M facts.
  • zipfile keeps one small entry record per document.
  • The operations log and curation-archive rows (data/*.json) are still read whole. They are usually small.
  • Export still loads all observations (and a fact-location index) into memory before writing observations.json. That needs its own follow-up.

Notes

  • Oracle: the new executemany calls go through the same rewrite and argument conversion as execute, but were not run against a real Oracle database.
  • A bank archive with a missing attachment entry is now reported when the restore runs, not at submission.
  • System tests test_52_transfer.py fail the same way on main locally (a background refresh_mental_model hits an unscripted stub rule), so they're unrelated.

Tests

  • New tests/test_transfer_streaming_import.py: JSON reader across read boundaries (split items, cut numbers, multi-byte text), truncated or non-array entries, referenced-id extraction from nested JSON text, and a full import from a file with batch size 1 that checks the hook still gets every document's text.
  • test_transfer_restore_rows_oracle.py: rows stream in batches per column set.
  • Existing transfer, clone and admin import suites pass.

Review fix: the history restore now reads rows one at a time and retargets
them as they stream; nothing exercised that path end to end.
@nicoloboschi
nicoloboschi merged commit 40cebcb into main Oct 16, 2026
117 checks passed
@nicoloboschi
nicoloboschi deleted the fix/import-streaming-5524 branch October 16, 2026 16:40
@nicoloboschi nicoloboschi mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import holds all observations in memory and writes them in one transaction

1 participant