Repository navigation
fix(transfer): import an archive without holding it in memory (#5524) - #5696
Merged
Merged
Conversation
Review fix: the history restore now reads rows one at a time and retargets them as they stream; nothing exercised that path end to end.
This was referenced Oct 20, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #5661. Closes #5524.
Import used to hold a bank in memory several times over: the whole uploaded zip, every parsed document, every observation, every document's text until the end (for the metering hook), and one remapped id per fact and observation. Now it reads the archive as it replays it.
What changed
retrieve_stream) and reads zip entries from there. The upload endpoints pass the spooled upload file instead ofawait file.read(); submission stores it withstore_stream.observations.json,history/audit_log.json,history/llm_requests.json) are decoded one array item at a time by a small incremental JSON reader, sojson.loadsno longer builds the whole list.(doc, ordinal) -> unit idmap is now one list per document.remapped_unit_idskeeps only the ids that mental models, their history, the graph queue and the curation archive actually cite, instead of every fact and observation.created_at,event_dateand source links with oneexecutemanyper statement, not up to 3 queries per observation._restore_rowswrites 500 rows perexecutemany.Measured
Synthetic archive with stubbed embeddings, peak Python memory (tracemalloc) for the whole import:
Peak memory now stays almost flat as the bank grows. Speed is about the same; tracemalloc makes the timings noisy.
What still grows with the bank
zipfilekeeps one small entry record per document.data/*.json) are still read whole. They are usually small.observations.json. That needs its own follow-up.Notes
executemanycalls go through the same rewrite and argument conversion asexecute, but were not run against a real Oracle database.test_52_transfer.pyfail the same way onmainlocally (a backgroundrefresh_mental_modelhits an unscripted stub rule), so they're unrelated.Tests
tests/test_transfer_streaming_import.py: JSON reader across read boundaries (split items, cut numbers, multi-byte text), truncated or non-array entries, referenced-id extraction from nested JSON text, and a full import from a file with batch size 1 that checks the hook still gets every document's text.test_transfer_restore_rows_oracle.py: rows stream in batches per column set.