Skip to content

Home cannot start, update or show Recovery with a file over 16 MiB (or 64 MiB total) in Library folders #444

Open

Description

@andersalm

Problem

A Home holding ordinary media cannot start, update or show Recovery status. Any single protected file over 16 MiB, or more than 64 MiB in total, makes startup fail with:

Error: declared principal-root object exceeds 16 MiB

The message names no file and no fix.

Reported (Sasha, 18 Oct, 0.8.0-alpha.22 on macOS via install.sh): he saved a 22.4 MB mp4 to Documents while Home was running, and afterwards Home would not restart. Moving the file out fixed it.

Cause (source at bfefe92; also in v0.7.1, auth.rs:5838)

  • auth.rs:76-77: MAX_MIGRATION_OBJECT_BYTES = 16 MiB and MAX_MIGRATION_TOTAL_BYTES = 64 MiB.
  • collect_declared_principal_root_objects (auth.rs:7042-7100) reads every declared object fully into memory and refuses on either limit. It applies to already-encrypted envelopes and plaintext alike, so the limits measure the serialized ciphertext, base64 overhead included.
  • The protected roots are the whole personal Library: Desktop, Documents, Pictures, Videos, Downloads, Public, .Trash and Library's runtime folder (documents.rs:723, library.rs:5741, api/auth_gateway.rs:1404). Moving a file to Trash does not help.
  • The check runs:
    • on every gateway start: gateway_cmd.rs:63,110 → verify_configured_principal_roots_ready → verify_declared_principal_roots_ready (auth.rs:7562) → preflight (auth.rs:7705);
    • during update: update.rs:1715; so the 0.7.1 → 0.8.0 re-install is affected;
    • for Recovery status: auth_gateway.rs:1272.
  • Library's chunked upload accepts files up to 100 MiB (gateway.rs:218, gateway_provider_proxy.rs:520-550), and nothing on the write path enforces the startup limits.

Required behavior (0.8.0 blocker)

  • A Home with large media starts, updates and shows Recovery status, with bounded memory. Test with: a 100 MiB file, many files totalling several GiB, a large file in Trash, and an old unencrypted file needing migration.
  • Authentication of protected objects stays exactly as strong: startup, update and Recovery status check headers and keys, and opening a file fully authenticates it before use (approved check-at-open design).
  • Old unencrypted files migrate within a memory bound: above a stated size, migration refuses with a named action (v3 scope); batched migration of large old files is Legacy-plaintext migration: batched migration with free-space admission and step-up timing #448.
  • Any remaining refusal names the file and the action that fixes it.
  • Installed proof on the release candidate, Mac and Linux, including the update path.

Not sufficient on its own: refusing large uploads. Existing Homes and data from 0.7.1 would still fail.

Notes

Found during alpha.23 acceptance. alpha.23 canary publication proceeds, since the defect is already in alpha.22 canary. 0.8.0 stable does not ship until the boxes above are done. Devil's-advocate check: MODIFY, keep as a stable blocker (high confidence).

Activity

  1. added this to the 0.8.0 milestone on Oct 18, 2026
  2. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Reproduced on the exact alpha.23 candidate source bfefe92f6: OMP's isolated source-home test Home on macOS, which has an owner principal; restarted with scripts/mac-source-home-restart.sh. Upgrade receipts are in the test Home's logs/principal-root-upgrade-20261018T1527*.json.

    Documents contents Restart result
    one 17 MiB file exit 1: declared principal-root object exceeds 16 MiB (Sasha's text)
    five 15 MiB files, 75 MiB total exit 1: declared principal-root objects exceed 64 MiB
    files removed exit 0, already_ready, Home serves

    The helper prints only "principal-root upgrade failed; rollback retained for reconciliation". The receipt names no file. The files were placed directly as plaintext; per the code (auth.rs:7042-7100), the same limits apply to encrypted envelopes written by Library uploads.

  3. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Independent read-only audit confirms the completed source-Home reproduction: the 17 MiB input produced "declared principal-root object exceeds 16 MiB" (53-byte output, SHA-256 e61e0e1e69ca86478d83948d5d0b5d2276fa88e2aee4fada37a76b287f310903); five 15 MiB inputs produced the 64 MiB total refusal (53 bytes, 860e69cfdc0c156c4b9de3210a31944c523848115f2dff1c786fe0a21afa92b0).
    Both removal controls returned restart exit 0 and the matching successful upgrade output (144 bytes, f4068408375196331a9d59091f778a1910bd58adf622ea351125cff07e1a8a84). This exercises the source-Home offline upgrade/restart path; signed installed update, Recovery status and complete storage acceptance remain open.

  4. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Design status (no code yet).

    • Draft design (OMP design agent): keep the envelope format; a streaming AES-GCM codec (hand-composed CTR+GHASH); a per-root verified record, so each object is fully authenticated once per version of its bytes. About 2,000 lines (estimate).
    • Source-first quality review: FAIL as the smallest correct fix.
      • Plaintext is only ever produced after full AES-GCM on read (auth.rs:5264-5296) and in migration round-trips (:7857-7908). Startup can check the envelope prefix and data_key_id binding (:5728, :7185-7206) in work proportional to file count, with no custom crypto.
      • It also requires in scope: Recovery status derived from the protection record instead of the 1 Hz object walk (gateway_home_system.rs:13, :5336); lifting the 1024-objects-per-root startup bound (auth.rs:73,7078,7154); damaged files shown per item instead of failing a whole listing (library.rs:3417-3427); migration encrypting once per object; one classifier for all sizes.
    • Devil's advocate: MODIFY (high confidence on code findings).
      • The wording must be honest: startup shows "no detected plaintext, headers bound to the current key, content integrity checked when opened". A forged header is caught at read time, not at startup.
      • Keep full authentication before deleting migration backups (auth.rs:8582-8601).
      • Migration of large legacy plaintext still needs a memory bound (library.rs:3667-3705, :3839-3859 have no cap).
      • Coordinate the Library part with Keep Library navigation, uploads and rendering coherent under load #428.
    • Pending: Anders's decision on amending the Done means to the honest startup guarantee, with full authentication at open. Irzhy reviews the security argument before any merge.
  5. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Anders decided (direct message to OMP, 2026-10-18 13:25 COT): "Yes on #444: check at open, with the devil's advocate corrections (honest wording, full check before deleting migration backups, memory bound for migrating large old files), Irzhy reviews before merge."

    Design direction, per local design notes 444-qr-own.md plus the devil's-advocate corrections:

    • Startup, update and Recovery status show, in work proportional to file count: no detected plaintext under a protected root; every envelope's data_key_id matches the root's current key; no migration journal pending. Recovery's wording says exactly that, with content integrity checked when a file is opened.
    • Opening a file fully authenticates it (AES-GCM) before any plaintext is used, as today. A damaged file shows per item with its name; it does not fail the whole listing.
    • Migration fully authenticates before deleting any backup, encrypts each object once, and has a memory bound for large legacy plaintext: it either streams or refuses above a stated size with a named action.
    • Also in scope: Recovery status derives from the protection record, with no 1 Hz object walk; the 1024-objects-per-root startup bound is lifted; one classifier for all sizes; every refusal names the file and the action.
    • Process: coordinate the Library part with Keep Library navigation, uploads and rendering coherent under load #428. Irzhy reviews the security argument before any merge.
  6. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Design v2: Codex cross-family review FAIL, with 6 blocking findings (/tmp/codex-runs/d444v2.final.md):

    1. A non-UTF-8 byte in a header makes the classifier return Plaintext, so damaged bytes get re-wrapped (auth.rs:7034).
    2. Deletion advice for a temp file that can be the only copy (auth.rs:9314-9322).
    3. The Home-state damage policy can silently reset state (gateway_home_system.rs:6650, :7713).
    4. Online migration admits protection before checking it can finish, plus the 180 s Recovery export token (auth_gateway.rs:1464, :1548).
    5. Backup directory names are not synced to their parents before commit, an inherited gap (auth.rs:9063, :7485).
    6. The slice order is unsafe.

    Two designs have failed review, so here is the assumption they shared: that #444 must also redesign legacy-plaintext migration and Home-wide damage handling. Findings 2-5 sit in those parts. Anders's approval allows "refuse above a stated size with a named action" as the migration memory bound, and existing migration already refuses above 16 MiB.

    Design v3 scope (narrowed, within the approved direction):

    1. A robust envelope classifier, independent of UTF-8 validity, where ambiguous means damaged and never plaintext.
    2. Startup, update and Recovery status classify by header only, with no size, total or count limits for envelopes, and the honest wording.
    3. Migration's commit and backup sequence is unchanged. Its limits apply only to the plaintext objects being migrated, and every refusal names the file and the action.
    4. Library shows a damaged envelope as Blocked per item.
    5. Split out as separate issues: the inherited backup-directory durability gap, the Home-state damage policy, and Library folder listing decrypts every file in the folder #445 listing cost.
  7. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Design v3 (/tmp/444-design-v3.md) follow-ups are filed (After 0.8.0; Irzhy may move any to 0.8.0): #446 crash safety (FU1, FU4), #447 Home-state damage policy (FU2), #448 batched legacy migration with space and step-up timing (FU3, FU7), #449 archive extraction bound (FU6), #450 Library open path race and memory (FU9, FU10). FU5 is #445. FU8, old updaters applying their own limits, goes in the alpha.24 release notes: Homes with large files re-run install.sh. v3 is in Codex cross-family review now.

  8. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Read the full cross-family review of design v3 at bfefe92: FAIL, with three blocking findings. The proposed classifier can treat damaged or valid envelopes as plaintext; its two read windows do not cover the stated JSON layouts; an unbounded activation scan can repeatedly expire Recovery export approval.
    OMP requested v4 at 20:12 UTC to revise the layout contract, conservative classification and approval order. Design review, implementation and installed Mac/Linux startup, update and Recovery proof remain open.
    The review accepts deferring the inherited Home-state and backup-directory durability work to #447 and #446 within this narrower scope, and confirms full authentication before backup cleanup remains in place. Irzhy’s required review and milestone decisions remain pending.

  9. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Design settled: v5 (~/.local/share/elastos-ops/444-design-v5.md, sha256 916f85a7…).

    Codex cross-family review, four rounds:

    • v2: FAIL, 6 blockers.
    • v3: FAIL, 3 blockers.
    • v4: FAIL, 1 false damage claim.
    • v5: FAIL, one remaining false claim, now corrected with Codex's own wording: "a full-length canonical envelope with at most five changed bytes is never classified Plaintext; it may remain Envelope until authentication at open".

    Codex confirmed in source:

    • no envelope that would authenticate can fail the full-parser re-check and be wrapped (auth.rs:5273, :7887);
    • wrapping preserves the exact bytes (:7848, :7906);
    • backup deletion keeps full authentication (:8582).

    The key contract (verified): every ElastOS writer since v0.7.1 emits one canonical pretty layout with ciphertext last (auth.rs:497-506, :5588, :7838; v0.7.1 :488-497, :4566, :6555).

    Implementation starts now on fix/444-check-at-open (worktree wt-444, from bfefe92f6), in the design's slice order:

    1. protection-record projection;
    2. classifier with damage at open and Library Blocked;
    3. approval before the scan;
    4. header-only readiness with plaintext-only limits and named refusals;
    5. wording and helpers.

    Then a Codex code review, Irzhy's security review, and alpha.24.

  10. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    The first #444 implementation task ended with S1–S3 committed. At 23:05 UTC, root verified unpublished HEAD 8ccdea618294fdfb7f85cfdecac6824ec422767b, tree fc42080a4596c3a8165299f62fe5ea93f12aa3ef; S4 still had tracked edits and a new test file, and S5 was unfinished.
    Root read the complete final report: S2 reported 3011 tests passed and 8 failed; seven “missing test binary env” failures are attributed to the environment by the author. S4's added tests remain unrun, and U18 remains unwritten, so the full verification gate stays open.
    Later owner evidence at 23:03 UTC launches Impl444b to finish S4/S5, correct doubled-path wording and run the required checks. Full implementation, code review, Irzhy's review and qualified installed acceptance remain open; alpha.24 is unqualified.

  11. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Design ↔ code reconciliation (implementation on fix/444-check-at-open; S1 66773ef33, S2 903ea66a8, S3 8ccdea618; S4/S5 in progress). These are deviations from design v5 accepted by OMP, so reviewers review one contract:

    1. The key-archive check runs only when the scan finds envelopes, as at the base auth.rs:6977. Unconditional checking broke Recovery status for protected roots without a kit archive.
    2. Rewritten test auth.rs:13543 follows §4.2: the offline upgrade never refuses a damaged file. "Refuse before the first root changes" is asserted with an over-limit plaintext file (P2) in a later root.
    3. U7/U13 assert bytes read (at most about 4 KiB + 3 per file) instead of a decrypt counter, which would be unreliable under parallel tests. No body read means nothing decrypted.
    4. Message format: a notice is the §7 sentence alone, because each sentence already starts with {path}. This removes the doubled "path: path …".
    5. Being fixed in S4: a replayed Kit export approval returned HTTP 500, because auth_error_response matches error text. It becomes a named 4xx with a test.

    These go to Irzhy with the design's §9/§12 review points.

  12. andersalm commented on Oct 18, 2026

    @andersalm
    ContributorAuthor

    Anders's decision (18 Oct): the #444 security review gate is replaced for alpha.24 (devil's advocate: MODIFY, conditions included):

    1. Before merge:
    2. Every serious finding is fixed before merge, and each finding gets a disposition tied to the merged head.
    3. Anders signs off Home cannot start, update or show Recovery with a file over 16 MiB (or 64 MiB total) in Library folders #444 for alpha.24 on that evidence. All other alpha.24 gates (0.8.0: Release 0.8.0 #93 6256550072) stay.
    4. Before 0.8.0 stable is signed: Irzhy, or another independent human Anders names, reviews Home cannot start, update or show Recovery with a file over 16 MiB (or 64 MiB total) in Library folders #444. A serious finding blocks 0.8.0.

    This replaces the earlier condition "Irzhy reviews before merge" (6253509891).

  13. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Reviews of the #444 implementation (fix/444-check-at-open, base bfefe92f6):

    • Codex code review at b749c6f8f (S1–S4): FAIL, 3 blocking.
      1. A Damaged candidate can exhaust the classification budget and cause a false P3 (auth.rs:7152).
      2. A symlink at declaration level stops Home with an unnamed refusal (auth.rs:7312).
      3. The "realtime stream opens no protected object" claim is false with a configured Profile: load_profile_authority, auth_gateway.rs:1400. Design §4.4 permits that lookup, so the claim gets corrected.
    • Codex delta review b749c6f8f..453f56eb2 (403 replay fix and S5): PASS.
    • Claude security-reviewer at 453f56eb2, source-first threat model: MERGE AFTER FIXES; no Critical or High findings.
      • Before merge: M1 (export and Profile setup refuse on Damaged files even when protection exists) and L5/L11 (control and bidi characters in path names reach terminal and helper output; a folder name can forge an Error: line).
      • Before publication: a Home-state damage check (FU2 newly reachable).
      • It independently found Codex 1 and 2 (its L2, L3), and a wording overclaim (L1: plaintext that looks like an envelope is left as is, so "no file is unencrypted" is false).

    Fix round started on 453f56eb2 (agent Fix444):

    • Codex 1–3, M1, L5/L11, honest wording (CHANGELOG entry and PROTECTED_CONTENT.md), and a gateway test for Home-state damage (FU2), each with a failing-before test.
    • Then a delta re-review by Codex and the security reviewer.

    Deferred to follow-ups (filed with the fix report; Irzhy sees them before stable): M3 (depth limits in writers and zip extraction; existed at base), L6–L10, and the S5 System residuals. Full report: /tmp/sec444.md (local).

  14. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Fix444 is the active correction round. Codex’s S1–S4 review remains FAIL, and the security review remains MERGE AFTER FIXES.
    Before alpha.24 publication, retain the security review’s installed Home-state damage check. The gateway test and the exact installed Home journey need separate receipts.
    The author reports 3,023 passed / 7 failed / 26 ignored for the whole server at 453f56eb2. The configured full gate and re-reviews remain open.

  15. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Fix round 453f56eb2..de2cf62765b11b5dd7d32d5902d28efd9a23cf9a has seven normal commits. The security delta verdict is MERGE; the separate Codex delta verdict is FAIL because refunding Damaged candidates removes the total bound on whole-file classification reads. This is a source-derived availability finding.
    OMP has assigned separate plaintext and probe-I/O budgets, named notices for candidates left unchecked, and the remaining System recovery messages. The default-and-preserve Home-state interim is committed; its installed journey remains required before alpha.24 publication. Both reviews must cover the corrected head.
    The author reports 3028 server tests passed, seven failed and 26 ignored; attribution of those failures to missing test-binary setup remains an author claim. Auth tests passed, but the Linux refusal case was skipped on Mac and branch release-note bounds still failed. The configured full gate, installed large-file/update/Recovery journeys and Anders’s #444 sign-off remain open.

  16. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Codex delta review PASS at 24b6dbe7df30fef39fbc70f9775d5c642100bb5c resolves the earlier whole-file probe-budget finding. The bound is 80 MiB per root scan plus per-file header/tail reads; total scan work still scales with file count.
    Security review at the same head says MERGE AFTER FIXES for the Unchecked recovery instruction and two Recovery-note claims. OMP assigned these text corrections at 01:41 UTC; their new commit and review coverage remain pending.
    These are source reviews. The full configured test gate, Anders's #444 sign-off, installed damaged-Home-state check before canary publication, and large encrypted-file startup/update/Undo/Recovery acceptance remain open.

  17. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Read the complete c444t2 review: PASS for 2b647a696, closing the c635 text failure. This review covers the text and matching expectation; the prior scan reviews retain their recorded scope.
    The author reported 3029 server tests passed, 7 failed and 26 ignored, and attributes the failures to missing test-binary environment. The integrated Linux gate is running.
    Anders’s #444 sign-off and the installed damaged-Home-state journey remain required before alpha.24 publication. Large encrypted-media, update/Undo, Recovery, memory and identity-continuity acceptance remain open.

  18. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Current public distribution check: release-head.json returns HTTP 200 and still names 0.8.0-alpha.23 (624 bytes, SHA-256 a93a9c3f01369bf57f9221dc8acde0cef0c1566e03a3a659c55cd61b08c7c072). Alpha.24 is unpublished, so running the public install command again now cannot fetch its fix.
    For an alpha.23 Home blocked by #444, recovery guidance must tie the reinstall to an available qualified fixed release. Preserve the Home's identity and data, and prove the affected installed upgrade/reinstall journey before describing it as a working repair.
    The existing alpha.23 canary approval remains unchanged. Alpha.24 qualification, publication authority and installed acceptance remain open.

  19. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    #444 review closure, branch fix/444-check-at-open head 2b647a696 (base bfefe92f6; 19 commits). Every commit is covered by a Codex cross-family review:

    Codex review Range Verdict Disposition
    c444s4 bfefe92f6..b749c6f8f FAIL, 3 blocking fixed in 257bf1565 (false P3), 78b13a0be (named link refusal), 9579a9462 (honest readiness claim; U14)
    c444s5 b749c6f8f..453f56eb2 PASS –
    c444fix 453f56eb2..de2cf6276 FAIL: the refund removed the aggregate probe bound fixed in b68c8bc31 (separate 80 MiB probe cap, new Unchecked kind; test probe_cap_names_unread_candidates_and_start_continues)
    c444cap de2cf6276..24b6dbe7d PASS –
    c444text 24b6dbe7d..c63592200 FAIL: Library re-upload advice false over 100 MiB fixed in 2b647a696
    c444t2 c63592200..2b647a696 PASS –

    Security review (Claude security-reviewer, source-first):

    • At 453f56eb2: MERGE AFTER FIXES.
      • M1 fixed in 2470dcd89 (test export_with_protection_returns_kit_despite_damaged_file).
      • L5/L11 fixed in 261f8363e (names_with_line_breaks_or_bidi_render_on_one_line, plus restart-smoke redaction).
      • L1 wording fixed in de2cf6276.
      • FU2 interim in 059bbd1a5 (damaged_home_state_file_leaves_shell_and_summaries_answering).
      • The S5 residuals were fixed in 24b6dbe7d.
    • At de2cf6276: MERGE.
    • At 24b6dbe7d: MERGE AFTER FIXES, text only. Fixed in c63592200: the Unchecked notice now says "other files", the start warning shows only with protection, and the 16 MiB sentence is in.

    Deliberate change from design §3.3, recorded as accepted by the security review: a candidate the scan cannot read within the 80 MiB probe cap is now named as Unchecked, and start continues. Before, it counted as plaintext and refused start. In the worst case, a genuine plaintext file that sits after ≥80 MiB of lookalike damaged files is only named, not protected, until the person moves the listed files. It can be resolved, because the notice says what to do and Library lazy protection encrypts the file when it is opened.

    Follow-ups: #447, #448, #449 (M3), #450 (comments 6257564191, 6257564415, 6257564664, 6257564947).

    Author checks at 24b6dbe7d: elastos-server --lib 3029 passed, plus the 7 environment-only "missing test binary env" tests; elastos-auth 25 passed; node tests pass.

    Integrated in the alpha.24 train release/train-0.8.0-rc9, with the Linux remote gate running. Irzhy, or a named human, reviews before 0.8.0 stable (6256668102).

  20. andersalm commented on Oct 19, 2026

    @andersalm
    ContributorAuthor

    Anders signs off #444 for alpha.24 (19 Oct, "444 signoff yes"). This is step 3 of the review substitution (6256668102).

  21. irzhywau commented on Oct 21, 2026

    @irzhywau
    Collaborator

    Irzhy's review of #444 as merged on develop: PASS on the three security points. Home calls a file unencrypted only when the parser that opens encrypted files also rejects its whole bytes, so a real encrypted file keeps its bytes. Backup cleanup is the same as before #444 and fully authenticates every migrated file before it deletes a backup. The Unchecked exception bounds start reads and names each file.
    The review does not include the Library Blocked display, the Kit approval order, the 403 replay fix or name rendering. Two race findings that existed before #444 go to #450.
    @andersalm, checkboxes 2 and 3 still say "no header-only shortcut" and "any size", which differs from your approved design. Please reword them to what is merged; the large unencrypted-file work continues in #448.

  22. andersalm commented on Oct 21, 2026

    @andersalm
    ContributorAuthor

    Thanks, Irzhy. I record your PASS for the three security points of #444 as merged on develop. Your comment names no commit. The commit 3beb81643 is my mapping: develop was at it when you reviewed and still is (alpha.24, the head of #375), and it contains all #444 commits. Step 4 of the 18 Oct review plan stays open. It closes when your #375 review names the frozen head (by Wed 28 Oct, as in #93).

  23. andersalm commented on Oct 22, 2026

    @andersalm
    ContributorAuthor

    SHR-048/049 disposition (22 Oct, OMP train owner) (#194 comment 6348425125): the receipts from the installed protected-content driver (scripts/protected-content-installed-e2e-proof.sh) are not used as acceptance evidence for 0.8.0-alpha.25 or 0.8.0.

  24. self-assigned this
    on Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions